[antlr-interest] Inherent lexer nondeterminism

Xue Yong Zhi zhixueyong at hotmail.com
Thu Mar 8 07:15:16 PST 2007

> my parser knows exactly when to expect FILE and when to expect SYMBOL, 
> but my lexer doesn't.

First, if possible, try to let the lexer maintains state itself (lookup 
tokens before and after, set a flag after special keywords etc), the 
decoupling between parser and lexer will make testing much easier.

But it may be impossible to do or too complicated, in that case, you 
have to let the parser set state for lexer. I did this in xruby's ruby 

Xue Yong Zhi
XRuby (Ruby to Java bytecode compiler):

More information about the antlr-interest mailing list