[antlr-interest] Conversion V2 to V3

Gavin Lambert antlr at mirality.co.nz
Thu Mar 20 05:35:18 PDT 2008


At 00:41 21/03/2008, David Wigg wrote:
>but when I replace this with {$type = HEXADECIMALINT;) it doesn't 
>work and I get warning (105)
>
>"No lexer rule corresponding to token: HEXADECIMALINT".

It should actually work despite giving that warning.

>tokens
>  {
>  OPERATOR = 'operator';
>  OCTALINT;
>  DECIMALINT;
>  HEXADECIMALINT;
>  FLOATONE;
>  FLOATTWO;
>  }

An odd quirk in recent versions of ANTLR makes it issue that 
warning when you've declared tokens like above and then tried to 
use them in the lexer.  (Supposedly the syntax above was designed 
only to introduce new virtual tokens for AST generation in the 
parser, but in earlier versions of ANTLR it did work in the lexer 
without problems.)

To get rid of the warning, just remove the tokens block 
declaration and add a fragment lexer rule instead:

fragment HEXADECIMALINT : ('0'..'9' | 'a'..'f' | 'A'..'F')+;

(Note that since this is a fragment rule and it's never referenced 
by any other lexer rule, the rule body itself is never actually 
invoked and so its contents don't actually matter.  They just 
can't be empty or the warning will come back again.)

It's a little clunky, admittedly, and I'm hoping the 
tokens-block-in-lexer syntax will come back (without the warning) 
in a future build.  But for the moment this is the way you need to 
go :)

Incidentally, you also might want to reorder the alts in your 
Number rule.  AFAIK the alts are tested in order, so you should 
have the less-specific prefixes (such as Digit) towards the end of 
the rule.



More information about the antlr-interest mailing list