[antlr-interest] Factoring token definitions

[Olivier Lefevre]

Sometimes a literal is used in several tokens and you want to factor it 
out. For instance you might want to define

   DIGIT: '0'..'9';
   INT: DIGIT+;

The problem with this is that when faced with a single digit the lexer will 
mark it as a DIGIT whereas you'd like to get an INT in all cases. Reversing 
the order of INT and DIGIT might give the desired results in the lexer, if 
I understand that correctly, but that fails even earlier, with ANTLR 
complaining that DIGIT is unreachable when you try to compile the grammar.

Basically, one would like to define DIGIT as a "non-token" token, i.e., a 
token literal that can be used in the definition of other token but cannot 
be matched as such by the lexer. I cannot find a way to do that...

Thanks,

-- O.L.