[antlr-interest] how to deal with the number and identifier tokens overlap problem?

John D. Mitchell johnm-antlr at non.net
Thu May 5 10:38:37 PDT 2005


>>>>> "Fan" == Fan Yang <fanyang_32 at yahoo.ca> writes:
[...]

> I encountered the number and identifier tokens overlap problem, for
> example, 1000-MAIN, the lexer will return INTEGER 1000 and a parsing
> error instead of "1000-MAIN" as IDENTIFIER.

Is there some reason that you're allowing identifiers to start with a
digit?

Anyways, in terms of the technique to handle this type of overlap, you
might want to check out the Java lexer. In particular the handling of '.'.

Hope this helps,
		John


More information about the antlr-interest mailing list