[antlr-interest] "Tokens"

[Johannes Luber]

Steve Bennett wrote:
> Hi all,
>   I was wondering if someone could explain exactly what the "tokens"
> keyword does. The book (4.4)  doesn't really explain how it differs
>>from a series of lexer rules.
> 
> For example, what's the difference between this:
> 
> tokens {
>   FOO='FOO';
> }
> 
> and just this:
> 
> FOO: 'FOO';
> 
> The book says that this construct "allows you to provide an alias for
> a string literal" - but how is that different to just defining a
> normal lexer rule?

In the case above the only advantage is that rules defined in the
tokens-command have a higher priority than the normal defined rules in
case of some ambiguity like between identifiers and keywords. You don't
have to take care of the order as in the "FOO: 'FOO';"-case, but you
can't use additional actions.

> Also, how exactly do normal string literals operate? Is there a
> difference between this:
> 
> r: foo '!';
> 
> and:
> 
> r: foo EXCLAMATION;
> EXCLAMATION: '!';
> 
> ?

For combined parsers you can use both definitions - but do not mix them
like

r: foo '!';
l: bar EXCLAMATION;
EXCLAMATION: '!';

'!' won't be translated to an EXCLAMATION, but to e.g. Token23, so you
have then a MismatchException for r or for l, depending on the priority
between EXCLAMATION and Token23. Personally I would use '!' instead
EXCLAMATION, as it seems clearer to me. In the case of separated parsers
you have to use the EXCLAMATION-style, as ANTLR won't allow literals in
pure parser or tree parser grammars. It would have to generate a second
lexer in that case and as only one lexer is used to split the input
stream, the second wouldn't be ever executed.

Johannes