[antlr-interest] "Tokens"

[Gavin Lambert]

At 06:40 19/11/2007, Johannes Luber wrote:
 >> minus: '-';
 >> hyphen: '-';
 >
 >Not with the lowercase names though. ;)

Actually yes; there's no way you can successfully do that with 
uppercase names (lexer rules).  The above is equivalent to this:

minus: MINUS;
hyphen: MINUS;
MINUS: '-';

which will work fine (though it's a bit unusual).  Writing the 
original as lexer rules, though, would result in two ambiguous 
tokens and things would break.

Incidentally, I prefer using explicitly named tokens (whether 
through rules or the tokens section); not only does this generate 
better output (you get MINUS instead of Token12) but it makes it 
easier to get a sense of the separate lexing stage that is going 
on, and it's harder to stuff things up.  I don't use literals 
anywhere in my parser for that reason.