[antlr-interest] lexing multiple literals to one token

[Ric Klaren]

Robert Anderson wrote:
> I want to lex two different (interchangeable) keywords into the same
> token.  I want to use the tokens {..} mechanism because I want both of
> these to be considered by a testLiterals=true identifier rule option. 
> How do I do this?  The following don't seem to work:

You can't with the tokens section, that one is moved to a hashtable/map.
So the following are all invalid syntax:

> tokens {
>    MYTOK="form1";
>    MYTOK="form2";
> }
> 
> generates a "redefinition" warning.
> 
> tokens {
>   MYTOK = "form1","form2";
> }
> 
> generates an "unexpected token: ," error.
> 
> tokens {
>   MYTOK = ("form1"|"form2");
> }
> 
> generates a "expecting STRING_LITERAL, found '('" error.
> 
> I cannot seem to find any real documentation on what is allowable in a
> tokens {...} block.
> 
> I would create a rule, but this ends up causing problems with
> ambiguity with my identifier rule.  I would use the "literals option"
> described in the documentation to get around this, but that appears to
> have been removed and replaced with the token {..} mechanism.

You can a move it in the id rule and do a manual check in action code
then do a $setType (in the tokens section use tokens { MYTOK; } or you
can override the method that does the literal check in the lexer (name
evades me but it should be obvious from the name.

Cheers,

Ric