[antlr-interest] Tokenising for context specific reserved words

Roshan James roshanj at google.com
Thu Jul 17 16:22:19 PDT 2008


Hello,

I am trying to parse a language where there are words that have keyword
status in only some contexts/rules. In any other context those words can be
used as identifiers. However the default behavior of the lexer is that it
will generate special tokens for these always. Is there some way to work
around this?

As an example consider the rule called options below:
*options: 'format' INTEGER
*
I define identifiers as:
*ID : LETTER (LETTER | DIGIT)*;
*
However when I do this, the lexer generates a special token that has type
'format'. Thus, in any other part of the grammar where I expect to parse the
input string 'format' as an identifier the parser complains.

The solution that comes to mind is to change the above rule to be
options: ID INTEGER
and then inserting an appropriate semantic check.

Is there a better way to do this?

regards,
Roshan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080717/3c8d9afc/attachment.html 


More information about the antlr-interest mailing list