[antlr-interest] Matching tokens only at certain places

[Terence Parr]

On Jun 19, 2006, at 1:53 PM, Emond Papegaaij wrote:
>> The gated predicate gates all sequences associated with that token  
>> in/
>> out.  It dynamically alters the prediction DFA to not see certain
>> paths.  This lets you turn off various tokens when a predicate is
>> false.   By default all tokens are visible with a {true}=> gated
>> predicate.
>
> In my case, I only have a single rule (well in my real parser I've  
> got 3) that
> conflicts with other rules. I was hoping I could gate that single  
> rule, to
> force the lexer to use the others. With the current behaviour, a  
> single rule
> like this
>   ALL_NONE: {false}?=> (.)*
> will effectively disable all rules in your entire lexer.
> In the resulting DFA all edges are
> removed, because all edges can possibly lead to the disabled token.

Not if you have another rule like

FOO : 'foo' ;

above it.  Or at least that is my intention ;)

> Isn't is
> possible to only remove those edges that lead to ALL_NONE? That  
> would disable
> ALL_NONE, but leave all other tokens intact. Of course when  
> ALL_NONE is
> re-enabled, you would have ambiguity, but that would be solved the  
> normal
> way, as if no predicates existed. Having to put predicates in all  
> lexer
> rules, just to be able to gate a single rule doesn't seem like an easy
> solution.

Hmm....well, let's look at a simple example:

lexer grammar t;

A : 'a' ;

B : 'b' ;

C : {sig}?=> ('a'|'b') ;  // match stuff here if sig

It generates the (somewhat weird) DFA for differentiating tokens:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: t_dec-1.png
Type: image/png
Size: 38183 bytes
Desc: not available
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20060619/eb89bcc1/t_dec-1-0001.png
-------------- next part --------------

That actually doesn't look right!  When sig is false, everything is  
turned off!  ack.  Ok, looks like another bug.  Wow!  Two  
bugs...you're the winner of the season! ;)

Let me dig into this...have you tried w/o the gate by the way?  just  
{sig}? might work.

Ter