[antlr-interest] Problems with lexing tokens containing blanks
Bernd Vogt
Bernd.Vogt at Innovations.de
Wed Nov 29 08:44:09 PST 2006
Hi all,
in my current project I have the requirement to lex some tokens like this:
lexer grammar ExampleLexer;
…
INDEX : 'index' ;
INDEX_OF : 'index of' ;
INT : '0' | '1'..'9' '0'..'9'* ;
Internally, the generated lexer does somethin like this:
if ('i' 'n' 'd' 'e' 'x' ' ') -> INDEX_OF
else if ('i' 'n' 'd' 'e' 'x') -> INDEX
Now I get a problem lexing the following input:
“index 4”
The lexer detects 'i' 'n' 'd' 'e' 'x' ' ' and expects the token
INDEX_OF. So the lexer throws an exception that he expects 'o' and
doesn't mach the Tokens INDEX and INT...
Hm, I think, I need to teach the lexer to do something like this:
if ('i' 'n' 'd' 'e' 'x' ' ' 'o') -> INDEX_OF
else if ('i' 'n' 'd' 'e' 'x') -> INDEX
How can I do that?
Kind regards
Bernd
P.S.: We are using ANTLR3.
More information about the antlr-interest
mailing list