[antlr-interest] ANother mismatched token 0!=0

[Gavin Lambert]

At 02:53 15/03/2008, G R wrote:
>IDMEFPATH
>     :    LETTER (LETTER | SCORIES);
>VALUE
>     :    (SCORIES | DIGITS | LETTER)+;
>DIGITS
>     :    DIGIT+;
>
>TAG
>     :    '@_' LITERAL;
>LITERAL
>     :    LETTER (LETTER | '-' | '_')*;
>
>fragment SCORIES
>     :    '-' | '_' | ':' | '.' | '?' | '!' | '|' | '@' | '#' | 
> '$' | '^' | '~' | '(' | ')' | '[' | ']' | '\\' | '/' | '*';
>fragment LETTER
>     :    ('a'..'z' | 'A'..'Z');
>fragment DIGIT
>     :    '0'..'9';
>
>NEWLINE
>     :    '\r'? '\n';
>WS
>     : (' '|'\t'|'\n'|'\r')+ {skip();};

These rules are going to give you a lot of problems, since you 
have a lot of overlaps in the public rules.

Remember that lexing is done first and context-free; all public 
rules are viable output tokens.  Given the excerpt above, "423" 
will *always* be a VALUE and not a DIGITS, since they're of 
equivalent length and the VALUE rule is listed first.  By 
contrast, sometimes you won't get NEWLINE tokens emitted since 
they may be swallowed up within a WS token.