[antlr-interest] Inherent lexer nondeterminism
Xue Yong Zhi
zhixueyong at hotmail.com
Thu Mar 8 07:15:16 PST 2007
>
> my parser knows exactly when to expect FILE and when to expect SYMBOL,
> but my lexer doesn't.
First, if possible, try to let the lexer maintains state itself (lookup
tokens before and after, set a flag after special keywords etc), the
decoupling between parser and lexer will make testing much easier.
But it may be impossible to do or too complicated, in that case, you
have to let the parser set state for lexer. I did this in xruby's ruby
grammar.
--
Xue Yong Zhi
XRuby (Ruby to Java bytecode compiler):
http://xruby.blogspot.com
More information about the antlr-interest
mailing list