[antlr-interest] Matching tokens only at certain places

[Emond Papegaaij]

On Monday 19 June 2006 20:00, Terence Parr wrote:
> On Jun 19, 2006, at 8:08 AM, Emond Papegaaij wrote:
> > On Monday 19 June 2006 16:13, Emond Papegaaij wrote:
> >> On Monday 19 June 2006 15:32, Emond Papegaaij wrote:
> >> This is what the resulting s0 DFA state in the generated code
> >> looks like:
> >>
> >> public DFA.State transition(IntStream input) throws
> >> RecognitionException {
> >>   int LA3_0 = input.LA(1);
> >>   if ( LA3_0=='{' && (sig)) {return s1;}
> >>   if ( LA3_0=='}' && (sig)) {return s2;}
> >>   if ( LA3_0=='i' && (sig)) {return s3;}
> >>   if ( LA3_0==';' ) {return s4;}
> >>   if ( (..)||(..)||(..) && (sig)) {return s5;}
> >>   if ( (..)||(..)||LA3_0==' ' && (sig)) {return s6;}
> >>   if ( (..)||..||(..)||(..)||(..)||(..)||..||(..) && (sig))
> >> {return s7;}
> >>   NoViableAltException nvae =
> >>     new NoViableAltException("", 3, 0, input);
> >>   throw nvae;
> >> }
> >>
> >> It is clear that this disables all paths except "LA3_0==';'" when
> >> 'sig' is false. As a result the lexer will only except ';' tokens
> >> as long as 'sig' is false. Am I using the {..}?=> predicates
> >> incorrectly? 

<cut bug in character comparison>
> Doh!  Consider me a moron.  Sorry about that...adding to bug fix list
> (well, will take 3 seconds to fix).
<cut solution>
> Sorry about that.

No problem, we are all humans right? :) Thanks for the quick-fix.

>
> > To come back to the first problem. Is it really preferable to let
> > the gated
> > semantic predicates disable all paths that /could/ lead to a certain
> > alternative? To me it seems more logical to let the predicates
> > remove all
> > paths that /will/ lead to a certain alternative. But I might be
> > missing
> > something obvious.
>
> The gated predicate gates all sequences associated with that token in/
> out.  It dynamically alters the prediction DFA to not see certain
> paths.  This lets you turn off various tokens when a predicate is
> false.   By default all tokens are visible with a {true}=> gated
> predicate.

In my case, I only have a single rule (well in my real parser I've got 3) that 
conflicts with other rules. I was hoping I could gate that single rule, to 
force the lexer to use the others. With the current behaviour, a single rule 
like this
  ALL_NONE: {false}?=> (.)*
will effectively disable all rules in your entire lexer. The syntax would 
suggest only that rule is disabled. In the resulting DFA all edges are 
removed, because all edges can possibly lead to the disabled token. Isn't is 
possible to only remove those edges that lead to ALL_NONE? That would disable 
ALL_NONE, but leave all other tokens intact. Of course when ALL_NONE is 
re-enabled, you would have ambiguity, but that would be solved the normal 
way, as if no predicates existed. Having to put predicates in all lexer 
rules, just to be able to gate a single rule doesn't seem like an easy 
solution.


Best regards,
Emond Papegaaij