[antlr-interest] tokens - when to use?
Juergen Weber
weberjn at gmail.com
Thu Oct 9 08:19:55 PDT 2008
Hi,
for the grammar below, when it parses the input
REMOVE NONE
there is a MismatchedSetException
When I change the NONE in r1 to NON the phrase is parsed.
Seems that the NONE is in a list of known tokens and cannot be entered
for ID, isn't it?
It surprises me, that using a TOKEN breaks a rule. It works after changing to
tokens { NONE;}
r1 : 'DELUSER' ('ALL' | NONE);
Can someony explain this behaviour? One should expect that r1 and r2
have a sufficently different prefix to make the parser know which rule
is current.
I didn't find anything on the web site nor in the book, the token
statement is not in the book, is it?
When should one use the tokens statement? Should every 'TOKEN' in
quotes be in the tokens section? This would be quite some kind of
work.
Thanks,
Juergen
grammar T;
options {
output=template;
}
r : r1 | r2;
r1 : 'DELUSER' ('ALL' | 'NONE');
r2 : 'REMOVE' id;
id: ID | QUOTEDSTRING;
QUOTEDSTRING
:
'\''
( ~('\'') )*
'\''
;
ID : ('0'..'9' | 'a'..'z'|'A'..'Z')+ ;
WS : (' '|'\t')+ { $channel=HIDDEN; } ;
More information about the antlr-interest
mailing list