[antlr-interest] Tokens

[David-Sarah Hopwood]

Ronald Sok wrote:
> Kevin J. Cummings wrote:
>> You are close.  What you have here is keywords as opposed to reserved
>> words.  When implementing the former, you will need to do something like
>> (at least this is what I do using ANTLR 2.7.7):
>>
>> id : ID
>>    | k:keyword
>>       { #k->setType(ID); }
>>       // This changes the token type of a keyword to an ID
>>    ;
>>
>> keyword
>>    : APPLE | PEAR | ORANGE
>>    ;
>>
>> someName
>>    :     'Name:' id NEWLINE
>>    ;
[...]
> Ok, I tried to change this into ANTLR 3 syntax, but ran into the fact that
> the result of keyword is a subtype of ParserRuleReturnScope, which
> definitely is not a Token type. So trying to change the type of the token
> like how you showed in ANTLR 2.7.7 syntax does not work for me.
> This is what I have, which results in an error while generating the grammar.
> 
> id : ID
>    | keyword
>       { $type=ID; }
>       // from http://www.antlr.org/wiki/display/ANTLR3/Migrating+from+ANTLR+2+to+ANTLR+3
>    ;

It's not necessary to change the type of the token unless you're actually
going to depend on that type in some other rule that uses 'id'. I.e. just
omitting '{ $type = ID; }' will work in most situtations.

If you do want to change it, then that can't be done in this case using
'$type =', which only works in a lexer rule. To change the type of the
token that is at the root of a particular subtree in a parser rule, use:

  id : ID
     | k=keyword { if ($k.tree != null) $k.tree.getToken().setType(ID); }
     ;

(Yes, it's a bit ugly. It is possible for $k.tree to be null in certain
error cases, unfortunately. If anyone has a way of doing this that is less
ugly but always works, I'd like to hear it.)

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20091128/51f05225/attachment.bin