[antlr-interest] Non-disjoint tokens

[Loring Craymer]

----- Original Message ----
> From: Steve Bennett <stevagewp at gmail.com>
> To: Gavin Lambert <antlr at mirality.co.nz>
> Cc: antlr-interest Interest <antlr-interest at antlr.org>
> Sent: Monday, November 26, 2007 4:44:43 AM
> Subject: Re: [antlr-interest] Non-disjoint tokens
> 
> On 11/26/07, Gavin Lambert  wrote:
> > I prefer this sort of approach over using semantic predicates; I
> > try to use those as little as possible.  (Mainly because I think
> > they're ugly, but also because they're target-specific.)
> 
> Thanks for the tip. I'm new to Antlr, but so far I'm finding a bit of
> a preference the other way, actually: I find syntactic predicates
> rather clunky and inexpressive, and since they're simply semantic
> predicates in disguise, they don't seem elegant to me. Most of the
> time they seem to end up in chains like this:

synpreds are not "simply sempreds in disguise"; that is a "feature" of the current implementation.  They are actually a lookahead device, and may be implemented (more) efficiently in some future version of ANTLR.

> 
> foo:
>   (bar) => bar
> | (dar) => dar
> | (lar) => lar
> | (rar) => rar
> ;
> 
> When all I really want to say is: "match the first expression that
> fits, and for god's sake don't give me any warnings about it!"

The usual behavior is "match the expression that fits"; adding "first" imposes order.  Synpreds impose that ordering.

> 
> There also seems not to be an easy way to say "in this context, don't
> try matching X, it won't work and you'll get stuck". Or for that
> matter a way of incorporating previously matched tokens: "If you have
> matched A and B, and the next token is D, then now would be a good
> time to match a C."

I suspect that this comment is lexer-specific, as is Gavin's comment about needing synpreds in places where they should not be needed.  This is on Ter's "to be fixed" list, and actually has to do with the use (or non-use, as happens now) of FOLLOW sets in lexers.

> 
> Both semantic and syntactic predicates seem like very powerful tools,
> but they seem a bit immature at the moment, and just need a bit more
> bedding down.

There is some truth to this, if you change the "they" to "the current implementation" in the "seem a bit immature" description.  Both PCCTS and ANTLR 2 provide strong bases of experience with predicates.  ANTLR 3 uses a different analysis approach (LL*), which is sufficiently powerful that Ter originally thought he could dispense with synpreds until experience showed otherwise.

The best way to get past this is to report use cases which "should" work but do not; that helps provide fodder for testing.

--Loring

> 
> Steve
> 




      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs