[antlr-interest] Conversion V2 to V3
Gavin Lambert
antlr at mirality.co.nz
Thu Mar 20 05:35:18 PDT 2008
At 00:41 21/03/2008, David Wigg wrote:
>but when I replace this with {$type = HEXADECIMALINT;) it doesn't
>work and I get warning (105)
>
>"No lexer rule corresponding to token: HEXADECIMALINT".
It should actually work despite giving that warning.
>tokens
> {
> OPERATOR = 'operator';
> OCTALINT;
> DECIMALINT;
> HEXADECIMALINT;
> FLOATONE;
> FLOATTWO;
> }
An odd quirk in recent versions of ANTLR makes it issue that
warning when you've declared tokens like above and then tried to
use them in the lexer. (Supposedly the syntax above was designed
only to introduce new virtual tokens for AST generation in the
parser, but in earlier versions of ANTLR it did work in the lexer
without problems.)
To get rid of the warning, just remove the tokens block
declaration and add a fragment lexer rule instead:
fragment HEXADECIMALINT : ('0'..'9' | 'a'..'f' | 'A'..'F')+;
(Note that since this is a fragment rule and it's never referenced
by any other lexer rule, the rule body itself is never actually
invoked and so its contents don't actually matter. They just
can't be empty or the warning will come back again.)
It's a little clunky, admittedly, and I'm hoping the
tokens-block-in-lexer syntax will come back (without the warning)
in a future build. But for the moment this is the way you need to
go :)
Incidentally, you also might want to reorder the alts in your
Number rule. AFAIK the alts are tested in order, so you should
have the less-specific prefixes (such as Digit) towards the end of
the rule.
More information about the antlr-interest
mailing list