[antlr-interest] Inherent lexer nondeterminism

[Xue Yong Zhi]

> 
> my parser knows exactly when to expect FILE and when to expect SYMBOL, 
> but my lexer doesn't.

First, if possible, try to let the lexer maintains state itself (lookup 
tokens before and after, set a flag after special keywords etc), the 
decoupling between parser and lexer will make testing much easier.

But it may be impossible to do or too complicated, in that case, you 
have to let the parser set state for lexer. I did this in xruby's ruby 
grammar.


-- 
Xue Yong Zhi
XRuby (Ruby to Java bytecode compiler):
http://xruby.blogspot.com