[antlr-interest] Re: Recommendation for Lexer

[Paul Johnson]

Martin Probst wrote:
>>>Plus it brings native support for the issues I have. 
>>
>>Can you tell me more about this? I can not see how it helps to maintain 
>>states.
> 
> 
> Well, you have a directive where you can enter normal states and
> explicit states, every rule can be preceded by a list of states in which
> it matches, or none (which means match in any except for explicit
> states). Also the lexer generates DFAs, so you don't get non-determinism
> problems because of tokens that only differ in suffixes (like ANTLR3,
> but that's just not ready...). 

[f]lex also does this but, in my experience, it's not generally useful 
if the parser can't communicate with the lexer and push back unnecessary 
tokens. For instance, if you're parsing with yacc/bison, then yacc 
already has one read-ahead token by the time you realise that you need 
to change states in the lexer, and it's too late. Have you got a fix for 
this? Or do you do all the necessary state-change logic in the lexer itself?

Paul