[antlr-interest] Inherent lexer nondeterminism

Omry Yadan omry at yadan.net
Fri Mar 9 04:00:38 PST 2007

Xue Yong Zhi wrote:

>> my parser knows exactly when to expect FILE and when to expect
>> SYMBOL, but my lexer doesn't.
> First, if possible, try to let the lexer maintains state itself
> (lookup tokens before and after, set a flag after special keywords
> etc), the decoupling between parser and lexer will make testing much
> easier.
I am not sure how to do this.
can you  provide an example please?

> But it may be impossible to do or too complicated, in that case, you
> have to let the parser set state for lexer. I did this in xruby's ruby
> grammar.

doing this will probably not fix the non-determinism warnings, right?

More information about the antlr-interest mailing list