[antlr-interest] greedy vs nongreedy lexer rules

Marcin Rzeźnicki marcin.rzeznicki at gmail.com
Sun Apr 18 16:26:29 PDT 2010


2010/4/19 Terence Parr <parrt at cs.usfca.edu>:
>
> On Apr 18, 2010, at 4:15 PM, Marcin Rzeźnicki wrote:
>> I am not sure whether it works: as far as I can remember I tried that
>> and, looking at generated code, I realized that GUTS was not using
>> 'follows' information so it didn't really know when to leave the loop.
>
> Hmm...it should since it's a fragment rule.

;-))
You replied:
The key to your question was:

"and it occurred to me that ANTLR does not really inspect follow set, it
seems to inspect only what's left in the rule itself."

That improvement/bug is what I intend to fix :)  Actually, ANTLR v4
lexers will just plain work better (i.e., more like what you expect)
;)



>
> Well, either way, we'd need to duplicate the GUTS then (nongreedy in v3 or modes in v4).
>
> Wow...this is a tough decision about how to do v4 lexers...I'd really like to avoid generating recursive-descent lexer code, sticking with a state machine.
>

The thing is that, as this example shows, you can completely get rid
of greedy/non-greedy by introducing, if you will, 'follow-awareness'.
I am not sure how hard it is, and how representative this example is,
but that's the conclusion I've come up with.

> Ter
>
>



-- 
Greetings
Marcin Rzeźnicki


More information about the antlr-interest mailing list