[antlr-interest] tokens - when to use?

[Martin Probst]

> Seems that the NONE is in a list of known tokens and cannot be entered
> for ID, isn't it?
>
> It surprises me, that using a TOKEN breaks a rule. It works after  
> changing to

This is due to the way Lexer and Parser separation works - the lexer  
produces tokens, which the parser consumes and uses to decide on which  
rule to take by their identifier. The token 'NONE' gets its own,  
distinct identifier, which is different from the identifier of your  
'ID' token.

The easiest way to work around this is to allow all tokens that are  
valid explicitly also as identifiers:

id: ID | QUOTEDSTRING | NONE | REMOVE | ...;

... which is a bit inconvenient as you'll have to add all the  
identifiers.

Your change probably wont work, your grammar will fail to recognize  
"DELUSER NONE".

HTH,
Martin