[antlr-interest] White spaces within token definition

Haralambi Haralambiev hharalambiev at gmail.com
Fri Apr 25 04:57:21 PDT 2008


I have stumbled upon a problem, that although has some workarounds, has
puzzled me over why it is happening.
(I searched for a similar question, but was unable to find it. I am sorry if
this has been answered somewhere else. If so, please provide me the link.)

Consider the following lexer grammar:
lexer grammar test;

ID : ('A'..'Z'|'a'..'z')+;
WhiteSpaces : (' '|'\t')+ {$channel=HIDDEN;};

Consider that the language that is recognized has many commands with the
syntax "COMMAND <name of the command>", but I am interested only in the exit
command, so I consider "COMMAND EXIT" as a token.
However, I would like
"COMMAND <something else>" to be matched as the sequence of two ID tokens.

With the grammar above, the "COMMAND EXIT" is successfully matched as
a CMD_EXIT token, however "COMMAND XYZ" produces an error "line
1:8 mismatched character 'X' expecting
'E'" and what is left (only the character Z) is matched as ID.

In the generated lexer class, in the mTokes() method I noticed that
the lexer will consider everything that starts with "COMMAND " as the
It just doesn't consider the characters in the token definition, that
were after the white space (i.e. 'E', 'X', 'I' and 'T') during the

So, if you could enlighten me on why is this happening, I will be very

Best Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080425/7289f246/attachment-0001.html 

More information about the antlr-interest mailing list