[antlr-interest] Non-disjoint tokens

[Steve Bennett]

On 11/27/07, Loring Craymer <lgcraymer at yahoo.com> wrote:
> > foo:
> >   (bar) => bar
> > | (dar) => dar
> > | (lar) => lar
> > | (rar) => rar
> > ;
> >
> > When all I really want to say is: "match the first expression that
> > fits, and for god's sake don't give me any warnings about it!"
>
> The usual behavior is "match the expression that fits"; adding "first" imposes order.  Synpreds impose that ordering.

Yes, but the syntax to make that happen is so inefficient. It would be
much nicer to be able to do:

foo options {matchfirst = true;} : bar | dar | lar | rar;

I'm finding a constant tradeoff between warning suppression and readability...

> I suspect that this comment is lexer-specific, as is Gavin's comment about needing synpreds in places where they should not be needed.  This is on Ter's "to be fixed" list, and actually has to do with the use (or non-use, as happens now) of FOLLOW sets in lexers.

Oh, goodie :)

> There is some truth to this, if you change the "they" to "the current implementation" in the "seem a bit immature" description.  Both PCCTS and ANTLR 2 provide strong bases of experience with predicates.  ANTLR 3 uses a different analysis approach (LL*), which is sufficiently powerful that Ter originally thought he could dispense with synpreds until experience showed otherwise.

Ah. Well, their uses seem pretty limited to me (suppressing warnings,
and preventing the parser from falling in holes), so I guess that's
why. It *would* be nice to be able to dispense with them.

> The best way to get past this is to report use cases which "should" work but do not; that helps provide fodder for testing.

Ok, I'll try and do that. So far, I'm probably not confident enough to
say what "should" work - all I can say is what I naively expected to
work. Is there a separate bugs/testing list, or do I just send them
here?

Steve