[antlr-interest] Tokens
Ronald Sok
ronald.sok at gmail.com
Fri Nov 27 19:12:32 PST 2009
David-Sarah Hopwood wrote:
> Kevin J. Cummings wrote:
>
>> On 11/27/2009 06:05 PM, Ronald Sok wrote:
>>
>>> Kevin J. Cummings wrote:
>>>
>>>> You are close. What you have here is keywords as opposed to reserved
>>>> words. When implementing the former, you will need to do something like
>>>> (at least this is what I do using ANTLR 2.7.7):
>>>>
>>>> id : ID
>>>> | k:keyword
>>>> { #k->setType(ID); }
>>>> // This changes the token type of a keyword to an ID
>>>> ;
>>>>
>>>> keyword
>>>> : APPLE | PEAR | ORANGE
>>>> ;
>>>>
>>>> someName
>>>> : 'Name:' id NEWLINE
>>>> ;
>>>>
>>>> You could reduce the number of productions by folding, but the principal
>>>> of changing the token type of keywords is what is important here. And
>>>> you may have to find out how to do this with ANTLR 3.x.
>>>>
>>> Ok, I tried to change this into ANTLR 3 syntax, but ran into the fact that
>>> the result of keyword is a subtype of ParserRuleReturnScope, which
>>>
>> Sorry, my bad, should be KEYWORD and done in the lexer, not the parser!
>>
>
> That won't work because either:
> - KEYWORD is before APPLE, PEAR and ORANGE, in which case it always takes
> priority and the type of a Token will never be APPLE, PEAR or ORANGE;
>
> - or, KEYWORD is after APPLE, PEAR and ORANGE, in which case those rules
> take priority and the type of a Token will never initially be KEYWORD.
> You could override it, but if you do that in a lexer rule then you
> don't have enough context to determine what it should be (and '$type ='
> can't be used in a parser rule).
>
> As I said in my other followup, it's usually not necessary to change the
> type (but you can do so using the code given in that post if you want).
>
>
Placing KEYWORD in the lexer indeed did not work for me for the reasons you
described. I agree that omitting {$type=ID;} and therefore not changing the
type of the token works well enough for me and my grammar.
Thanks all for helping me out.
Ronald
More information about the antlr-interest
mailing list