[antlr-interest] "lexer state" issues and equivalence to nextToken

[Ric Klaren]

On Fri, Aug 20, 2004 at 02:14:26AM -0000, Eric Mahurin wrote:
> First of all, if you have a lexer with these rules:
>
> X: <rule for X> ;
> Y: <rule for Y> ;
>
> should this be equivalent to:
>
> NextToken
> : <rule for X> {$setType(X);}
> | <rule for Y> {$setType(Y);}
> ;
> protected X: ;
> protected Y: ;

That should be equivalent.

> I have cases where X and Y look ambiguous, but that is because they
> are valid in separate "lexer states" (I may be using a this term in a
> different context than the docs talk about).  I qualify them with
> semantic predicates correlating to the mode:
>
> X: {cond}? ... ;
> Y: {!cond}? ... ;

I'm not one of the hoisting experts around ;) But I think you'd need it for
this way of writing it down. E.g. it won't work with ANTLR 2.7.4 since the
predicates won't get elevated to the nextToken method (check the generated
code)

> Unfortunately, this doesn't work and I have to resort to the NextToken
> form for the ones it complains about and this works.  Is this a bug?

Not directly a bug, but exactly pretty no.

> Is there a better way to handle these lexer modes?  I think I could
> spawn off a new lexer and use a selector for different modes, but I
> really don't want to deal with that complexity or overhead.

If you want truly lexer states then you'll have to use two lexers and a
selector. The few places where I needed it I found the overhead not that
big, but I guess it depends a bit on the complexity of the lexer, mine are
usually as simple as possible.

Cheers,

Ric
--
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++-------
   Words fly like arrows
      as if we knew what was right and wrong. --- Chuang Tsu



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/