[antlr-interest] Re: dfa-based lexers versus top-down antlr lexers

Terence Parr parrt at jguru.com
Tue Apr 29 09:44:07 PDT 2003


I'm thinking that a future version of ANTLR would pretty much support 
whatever "module" you wanted.  Sometimes a DFA-based lexer is easiest 
and sometimes a top-down lexer is better (such as parsing nested syntax 
within HTML tags or whatever).  Naturally you can always hook in your 
own hand-built lexer.

Thanks for the feedback.

On Monday, April 28, 2003, at 03:39  PM, Oliver Zeigermann wrote:

> I see what you mean. Lately a friend of mine asked me how ANTLR
> decides which token rule to use for a production. This made me talk
> for half an hour but I could not really get it explained. He was
> thinking of something like "the most specific rule will be the one
> to match" and I was trying to say "no, it is all very
> deterministic". Anyway, "most specific" is similar to what you made
> up using left factoring or how you called it "combine left
> edges". "Most specific" seems to be the most natural way to
> interpret token definitions. One thing to notice there is
>
> DIGIT1 : '1';
>
> is also more specific than
>
> INT : ('0'..'9')+ ;
>
> which can not be handeled with left factoring as it seems to me. For
> certain rule based systems it is instead very possible to make this
> distinction. How do they do this?!

I believe they use the first found rule.  Most specific has to be first 
if you want things to work right.

Ter
--
Co-founder, http://www.jguru.com
Creator, ANTLR Parser Generator: http://www.antlr.org
Co-founder, http://www.peerscope.com link sharing, pure-n-simple
Lecturer in Comp. Sci., University of San Francisco


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 




More information about the antlr-interest mailing list