[antlr-interest] Overlapping token definitions

[Jan van Mansum]

Thanks Raphael! Actually I tried that but had concluded that the order of
the token definitions did not matter.
However, you are right, it does matter. One FORMAT wasn't matched because I
forgot to make the
second part of it optional (left out the question mark):

fragment
ALPHANUM
    :    ('A'|'U') NUMBER ('A' NUMBER)?
    ;

Good point about keywords inside strings. I hadn't considered that and it
did break my parser. I now
fixed that by changing the order of the lexer rules as well. The order is
now:

FORMAT
DP_STRING
<keywords>

Thanks again,

regards,

Jan.


2008/7/18 <r_reitzi at cs.uni-kl.de>:

> Hi Jan!
>
> The problem is that your "special"  keywords are perfectly matched by rule
> DP_STRING (as well as FORMAT). But, obviously, lexer cannot _know_ which
> rule you want it to match in which case.
> If you want to interpret strings matching FORMAT _always_ as FORMAT, place
> that rule above DP_STRING. ANTLR's lexer chooses the first rule that
> matches.
> However, you should perhaps not assume that nobody enters a string in a
> field looking the same as your keywords. To see if this does disturb your
> parsing, the parser gramar must be taken into account. It depends on how you
> take context into account.
> Can you choose another symbol as keyword delimiter?
>
> Regards
>
> Raphael
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080718/f7e90481/attachment.html