[antlr-interest] Re: Recommendation for Lexer
Paul Johnson
gt54-antlr at cyconix.com
Thu Feb 9 01:07:58 PST 2006
Martin Probst wrote:
>>>Plus it brings native support for the issues I have.
>>
>>Can you tell me more about this? I can not see how it helps to maintain
>>states.
>
>
> Well, you have a directive where you can enter normal states and
> explicit states, every rule can be preceded by a list of states in which
> it matches, or none (which means match in any except for explicit
> states). Also the lexer generates DFAs, so you don't get non-determinism
> problems because of tokens that only differ in suffixes (like ANTLR3,
> but that's just not ready...).
[f]lex also does this but, in my experience, it's not generally useful
if the parser can't communicate with the lexer and push back unnecessary
tokens. For instance, if you're parsing with yacc/bison, then yacc
already has one read-ahead token by the time you realise that you need
to change states in the lexer, and it's too late. Have you got a fix for
this? Or do you do all the necessary state-change logic in the lexer itself?
Paul
More information about the antlr-interest
mailing list