[antlr-interest] "Tokens"
Johannes Luber
jaluber at gmx.de
Sun Nov 18 03:22:35 PST 2007
Steve Bennett wrote:
> Hi all,
> I was wondering if someone could explain exactly what the "tokens"
> keyword does. The book (4.4) doesn't really explain how it differs
>>from a series of lexer rules.
>
> For example, what's the difference between this:
>
> tokens {
> FOO='FOO';
> }
>
> and just this:
>
> FOO: 'FOO';
>
> The book says that this construct "allows you to provide an alias for
> a string literal" - but how is that different to just defining a
> normal lexer rule?
In the case above the only advantage is that rules defined in the
tokens-command have a higher priority than the normal defined rules in
case of some ambiguity like between identifiers and keywords. You don't
have to take care of the order as in the "FOO: 'FOO';"-case, but you
can't use additional actions.
> Also, how exactly do normal string literals operate? Is there a
> difference between this:
>
> r: foo '!';
>
> and:
>
> r: foo EXCLAMATION;
> EXCLAMATION: '!';
>
> ?
For combined parsers you can use both definitions - but do not mix them
like
r: foo '!';
l: bar EXCLAMATION;
EXCLAMATION: '!';
'!' won't be translated to an EXCLAMATION, but to e.g. Token23, so you
have then a MismatchException for r or for l, depending on the priority
between EXCLAMATION and Token23. Personally I would use '!' instead
EXCLAMATION, as it seems clearer to me. In the case of separated parsers
you have to use the EXCLAMATION-style, as ANTLR won't allow literals in
pure parser or tree parser grammars. It would have to generate a second
lexer in that case and as only one lexer is used to split the input
stream, the second wouldn't be ever executed.
Johannes
More information about the antlr-interest
mailing list