[antlr-interest] Resolving ambiguities in Lexer rules

Achint Mehta achintmehta at gmail.com
Sun Aug 16 10:06:31 PDT 2009


Hi David/Sarah,

Thanks for your response.

So If I have to include all the tokens in SPECIAL_STRING, then I assume that
SPECIAL_STRING has to be changed from lexer to parser rule.

Based on your and Joe's suggestion I have modified the grammar as following:
----------------------------------------------------------
grammar sample_parser;

options
{
   language=C;
}
tokens{
T_VERSION='VERSION';
T_COUNT='COUNT';
}

requestline : T_VERSION '=' special_string (NEWLINE)?;

special_string:WORD | tokenset;
tokenset: T_VERSION | T_COUNT;
fragment
CHAR: (('a'..'z')|('A'..'Z'));
fragment
DIGIT: ('0'..'9');
WORD: (CHAR|DIGIT)+;
NEWLINE:('\r')? '\n';
------------------------------------------------------------

This grammar seems to work.

Do you think this grammar is fine or can it be improved further ?

Thanks.

Regards,
Achint

On Sun, Aug 16, 2009 at 10:45 AM, David-Sarah Hopwood <
david-sarah at jacaranda.org> wrote:

> Achint Mehta wrote:
> > 2. The second option is that all the tokens have to given as alternate
> > rules/token with SPECIAL_STRING. Again, in a big/complicated parser, all
> the
> > tokens in the whole parser have to be repeated where-ever I intend to use
> > the SPECIAL_STRING. This can be simplified if I give the tokens in the
> > definition of SPECIAL_STRING iteself. But still in a parser which could
> use
> > tens or hundreds of tokens, it would seem to be impractical to repeat all
> > the tokens in SPECIAL_STRING rule and other similar rules (intended for
> > collecting the generic string).
>
> You only need one rule that collects all of the tokens, which can then be
> included as an alternative in SPECIAL_STRING and similar rules. This is
> the approach I use for ECMAScript 5 (which has some contexts in which
> keywords are treated like identifiers). The duplication is mildly
> irritating, but it still works quite well in practice.
>
> --
> David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090816/a2ff10c3/attachment.html 


More information about the antlr-interest mailing list