[antlr-interest] Re: Recommendation for Lexer

[Martin Probst]

> [f]lex also does this but, in my experience, it's not generally useful 
> if the parser can't communicate with the lexer and push back unnecessary 
> tokens. For instance, if you're parsing with yacc/bison, then yacc 
> already has one read-ahead token by the time you realise that you need 
> to change states in the lexer, and it's too late. Have you got a fix for 
> this? Or do you do all the necessary state-change logic in the lexer itself?

I'm doing all the state stuff in the Lexer, exactly because of that.
I've made bad experiences with setting Lexer state from the Parser, it
just makes the code messy and hard to debug. The major pro of JFlex here
is that it manages the states for you and includes that
explicit/non-explicit states thing. It's all possible to achieve in
ANTLR, but it gets really ugly (e.g. one big NEXT_TOKEN rule with a
predicates for each token, set types etc.).

Martin