[antlr-interest] Inherent lexer nondeterminism

[Omry Yadan]

Xue Yong Zhi wrote:

>
>>
>> my parser knows exactly when to expect FILE and when to expect
>> SYMBOL, but my lexer doesn't.
>
> First, if possible, try to let the lexer maintains state itself
> (lookup tokens before and after, set a flag after special keywords
> etc), the decoupling between parser and lexer will make testing much
> easier.
I am not sure how to do this.
can you  provide an example please?

>
> But it may be impossible to do or too complicated, in that case, you
> have to let the parser set state for lexer. I did this in xruby's ruby
> grammar.

doing this will probably not fix the non-determinism warnings, right?