[antlr-interest] antlr-interest Digest, Vol 47, Issue 9

[Juergen Weber]

>
> From: Martin Probst <mail at martin-probst.com>
> Subject: Re: [antlr-interest] tokens - when to use?
>
> The easiest way to work around this is to allow all tokens that are
> valid explicitly also as identifiers:
>
> id: ID | QUOTEDSTRING | NONE | REMOVE | ...;
>
> ... which is a bit inconvenient as you'll have to add all the
> identifiers.
>
> Your change probably wont work, your grammar will fail to recognize
> "DELUSER NONE".
>

No, DELUSER NONE indeed works, I checked in ANTLRWorks. And having to
add lots of Tokens to id is very impractical.

Yes, I saw in the debugger that the token IDs did not match. But why
would 'NONE' get a token ID and why would ANTLR try to match the ID?

Naively I'd expect something like if match('REMOVE') then
match(('0'..'9' | 'a'..'z'|'A'..'Z')+ ). Why should rule r2 have any
relation to the 'NONE' ID?

And, please, is there a FAQ or chapter in the book which explains this
and the token section?

Thanks,
Juergen