[antlr-interest] tokens - when to use?

Juergen Weber weberjn at gmail.com
Thu Oct 9 08:19:55 PDT 2008


Hi,

for the grammar below, when it parses the input

REMOVE NONE

there is a MismatchedSetException

When I change the NONE in r1 to NON the phrase is parsed.

Seems that the NONE is in a list of known tokens and cannot be entered
for ID, isn't it?

It surprises me, that using a TOKEN breaks a rule. It works after changing to

tokens { NONE;}
r1        :        'DELUSER' ('ALL' | NONE);


Can someony explain this behaviour? One should expect that r1 and r2
have a sufficently different prefix to make the parser know which rule
is current.

I didn't find anything on the web site nor in the book, the token
statement is not in the book, is it?
When should one use the tokens statement? Should every 'TOKEN' in
quotes be in the tokens section? This would be quite some kind of
work.

Thanks,
Juergen


grammar T;

options {
    output=template;
}


r         :         r1 | r2;


r1        :        'DELUSER' ('ALL' | 'NONE');

r2        :        'REMOVE' id;


id: ID | QUOTEDSTRING;


QUOTEDSTRING
    :
      '\''
      ( ~('\'') )*
      '\''
        ;

ID        :        ('0'..'9' | 'a'..'z'|'A'..'Z')+ ;

WS  :   (' '|'\t')+ { $channel=HIDDEN; } ;


More information about the antlr-interest mailing list