[antlr-interest] Unnecessary Java output file diffs

[Gavin Lambert]

At 04:55 28/05/2009, Jim Idle wrote:
>The idea is that a token is defined as specific type, but can 
>also exist as another type at the same time. So, if the parser 
>needs to know if there is an ID, the code generated asks if the 
>token is a real ID OR has a superimposed ID and so reduces all 
>the complexity of using keywords as identifiers (and probably 
>other case, but this is the specific problem domain), to an extra 
>|| operation and an additional LA type operation, something like 
>this:
>
>int LA1_0 = input.LA(1);
>int LA1_0_s = input.LAS(1);
>
>if ( (LA1_0==ID)||(LA1_0_s==ID ) {
>                 alt1=1;
>             }

Shouldn't any number of other token types be able to be 
superimposed on a token?  It seems like a waste to arbitrarily 
restrict this to one other type.

(In terms of implementation, you could make LAS return a BitSet of 
the possible types, or you could leave the decision-making to the 
token itself; eg. use TokenType instead of int and have it define 
comparison and containment operations.  Or something else along 
those lines.)