Previous Contents

6.7   Module Token: tokens for grammars

type t = string * string;;
type pattern = string * string;;
Token and token patterns. Token are build by the lexer. Token patterns come from the EXTEND statement.
The first string is the constructor name (must start with an uppercase character). When it is empty, the second string is supposed to be a keyword.
The second string is the constructor parameter. Empty if it has no parameter.
The way tokens pattern are interpreted to parse tokens is done by the lexer, function tparse below.
exception Error of string;;
An lexing error exception to be used by lexers.
type location_function = int -> int * int;;
The type for a function associating a number of a token in a stream (starting from 0) to its source location.
type lexer =
  { func : char Stream.t -> t Stream.t * location_function;
    using : pattern -> unit;
    removing : pattern -> unit;
    tparse : pattern -> (t Stream.t -> string) option;
    text : pattern -> string }
;;
The type for a lexer used by Camlp4 grammars.
The field func is the lexer function. The character stream is the input stream to be lexed. The result is a pair of a tokens stream and a location function for this tokens stream.
The field using is a function telling the lexer that the grammar uses this token (pattern). The lexer can check that its constructor is correct, and interpret some kind of tokens as keywords (to record them in its tables). Called by EXTEND statements.
The field removing is a function telling the lexer that the grammar does not uses the given token (pattern) any more. If the lexer has a notion of "keywords", it can release it from its tables. Called by DELETE_RULE statements.
The field tparse is a function returning either Some f, f being the parsing function associated with the pattern, or None if it is a standard parsing of the pattern.
The field text returns the name of some token pattern, used in error messages.

Previous Contents