[antlr-interest] custom TokenLabelType and EOF/Error tokens

[David-Sarah Hopwood]

David-Sarah Hopwood wrote:
> David-Sarah Hopwood wrote:
>> If you are subclassing CommonTokenStream or TokenRewriteStream,
>> I think it should be sufficient to override the LT method as follows:
>>
>>    protected static MyToken MY_EOF_TOKEN = new MyToken(CharStream.EOF);

Make this 'public static final' in order for the nextToken() code below
to work.

>>    @Override public Token LT(int k) {
>>      Token t = super.LT(k);
>>      return t != Token.EOF_TOKEN ? t : MY_EOF_TOKEN;
>>    }
>>
>> (The EOF_TOKEN doesn't actually exist in the token stream; it is
>> returned only when you look ahead using LT.)
>>
>> However, I haven't tested this,
> 
> I've tested it now; it appears to work.

I spoke too soon -- there is another method that needs to be overridden
in the lexer:

@lexer::members {
  @Override public Token nextToken() {
    Token t = super.nextToken();
    return t != Token.EOF_TOKEN ? t : MyTokenStream.MY_EOF_TOKEN;
  }
}

This is getting too complicated. Just to ensure that all tokens are of
the overridden type, we have to:

 - set the TokenLabelType option
 - override emit() and nextToken() in the lexer
 - override getMissingSymbol() in the parser
 - override LT() in the token stream
 - upgrade to ANTLR 3.2

and this is all quite fragile and possibly dependent on the Java target.
It clearly calls for a redesign of the way TokenLabelType is handled.

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20091111/d825944c/attachment.bin