6.7 Module Token: tokens for grammars
type t = string * string;;
type pattern = string * string;;
-
Token and token patterns. Token are build by the lexer. Token
patterns come from the EXTEND statement.
The first string is the constructor name (must start with
an uppercase character). When it is empty, the second string
is supposed to be a keyword.
The second string is the constructor parameter. Empty if it
has no parameter.
The way tokens pattern are interpreted to parse tokens is
done by the lexer, function tparse below.
exception Error of string;;
-
An lexing error exception to be used by lexers.
type location_function = int -> int * int;;
-
The type for a function associating a number of a token in a stream
(starting from 0) to its source location.
type lexer =
{ func : char Stream.t -> t Stream.t * location_function;
using : pattern -> unit;
removing : pattern -> unit;
tparse : pattern -> (t Stream.t -> string) option;
text : pattern -> string }
;;
-
The type for a lexer used by Camlp4 grammars.
The field func is the lexer function. The character stream is the
input stream to be lexed. The result is a pair of a tokens stream
and a location function for this tokens stream.
The field using is a function telling the lexer that the grammar
uses this token (pattern). The lexer can check that its constructor
is correct, and interpret some kind of tokens as keywords (to record
them in its tables). Called by EXTEND statements.
The field removing is a function telling the lexer that the grammar
does not uses the given token (pattern) any more. If the lexer has a
notion of "keywords", it can release it from its tables. Called by
DELETE_RULE statements.
The field tparse is a function returning either Some f, f being
the parsing function associated with the pattern, or None if it
is a standard parsing of the pattern.
The field text returns the name of some token pattern,
used in error messages.