[antlr-interest] Tokens

Kevin J. Cummings cummings at kjchome.homeip.net
Fri Nov 27 18:17:39 PST 2009


On 11/27/2009 08:39 PM, David-Sarah Hopwood wrote:
> Kevin J. Cummings wrote:
>> On 11/27/2009 06:05 PM, Ronald Sok wrote:
>>> Kevin J. Cummings wrote:
>>>> You are close.  What you have here is keywords as opposed to reserved
>>>> words.  When implementing the former, you will need to do something like
>>>> (at least this is what I do using ANTLR 2.7.7):
>>>>
>>>> id : ID
>>>>    | k:keyword
>>>>       { #k->setType(ID); }
>>>>       // This changes the token type of a keyword to an ID
>>>>    ;
>>>>
>>>> keyword
>>>>    : APPLE | PEAR | ORANGE
>>>>    ;
>>>>
>>>> someName
>>>>    :     'Name:' id NEWLINE
>>>>    ;
>>>>
>>>> You could reduce the number of productions by folding, but the principal
>>>> of changing the token type of keywords is what is important here.  And
>>>> you may have to find out how to do this with ANTLR 3.x.
>>>
>>> Ok, I tried to change this into ANTLR 3 syntax, but ran into the fact that
>>> the result of keyword is a subtype of ParserRuleReturnScope, which
>>
>> Sorry, my bad, should be KEYWORD and done in the lexer, not the parser!
> 
> That won't work because either:
>  - KEYWORD is before APPLE, PEAR and ORANGE, in which case it always takes
>    priority and the type of a Token will never be APPLE, PEAR or ORANGE;
> 
>  - or, KEYWORD is after APPLE, PEAR and ORANGE, in which case those rules
>    take priority and the type of a Token will never initially be KEYWORD.
>    You could override it, but if you do that in a lexer rule then you
>    don't have enough context to determine what it should be (and '$type ='
>    can't be used in a parser rule).

Hmmm, in the context I used it in, I had to be able to tell when a
keyword was being used as an identifier.  In ANTLR-2, I could override
the token's type in the parser.  (Makes expression evaluation a whole
lot easier further down the line when checking for an identifier....)

> As I said in my other followup, it's usually not necessary to change the
> type (but you can do so using the code given in that post if you want).

-- 
Kevin J. Cummings
kjchome at rcn.com
cummings at kjchome.homeip.net
cummings at kjc386.framingham.ma.us
Registered Linux User #1232 (http://counter.li.org)


More information about the antlr-interest mailing list