[antlr-interest] Re: Recommendation for Lexer

Martin Probst mail at martin-probst.com
Wed Feb 8 14:47:19 PST 2006


> > Plus it brings native support for the issues I have. 
> 
> Can you tell me more about this? I can not see how it helps to maintain 
> states.

Well, you have a directive where you can enter normal states and
explicit states, every rule can be preceded by a list of states in which
it matches, or none (which means match in any except for explicit
states). Also the lexer generates DFAs, so you don't get non-determinism
problems because of tokens that only differ in suffixes (like ANTLR3,
but that's just not ready...). 

So for this it provides any support I can think of, and the explicit/non
explicit thing is very useful if you have some language parts that are
whitespace significant. It's a little less convenient than ANTLR because
it doesn't impose any token model on you, so it also doesn't
automatically create a table of token types and you have to set types
manually in Java code. However, I can live with that in return for
states.

Martin



More information about the antlr-interest mailing list