[antlr-interest] Problems with lexing tokens containing blanks

Bernd Vogt Bernd.Vogt at Innovations.de
Wed Nov 29 08:44:09 PST 2006


Hi all,

in my current project I have the requirement to lex some tokens like this:

lexer grammar ExampleLexer;
…
INDEX : 'index' ;
INDEX_OF : 'index of' ;
INT : '0' | '1'..'9' '0'..'9'* ;

Internally, the generated lexer does somethin like this:

if ('i' 'n' 'd' 'e' 'x' ' ') -> INDEX_OF
else if ('i' 'n' 'd' 'e' 'x') -> INDEX

Now I get a problem lexing the following input:

“index 4”

The lexer detects 'i' 'n' 'd' 'e' 'x' ' ' and expects the token 
INDEX_OF. So the lexer throws an exception that he expects 'o' and 
doesn't mach the Tokens INDEX and INT...

Hm, I think, I need to teach the lexer to do something like this:

if ('i' 'n' 'd' 'e' 'x' ' ' 'o') -> INDEX_OF
else if ('i' 'n' 'd' 'e' 'x') -> INDEX

How can I do that?

Kind regards
Bernd

P.S.: We are using ANTLR3.



More information about the antlr-interest mailing list