[antlr-interest] custom TokenLabelType and EOF/Error tokens
David-Sarah Hopwood
david-sarah at jacaranda.org
Wed Nov 11 15:37:50 PST 2009
David-Sarah Hopwood wrote:
> David-Sarah Hopwood wrote:
>> If you are subclassing CommonTokenStream or TokenRewriteStream,
>> I think it should be sufficient to override the LT method as follows:
>>
>> protected static MyToken MY_EOF_TOKEN = new MyToken(CharStream.EOF);
Make this 'public static final' in order for the nextToken() code below
to work.
>> @Override public Token LT(int k) {
>> Token t = super.LT(k);
>> return t != Token.EOF_TOKEN ? t : MY_EOF_TOKEN;
>> }
>>
>> (The EOF_TOKEN doesn't actually exist in the token stream; it is
>> returned only when you look ahead using LT.)
>>
>> However, I haven't tested this,
>
> I've tested it now; it appears to work.
I spoke too soon -- there is another method that needs to be overridden
in the lexer:
@lexer::members {
@Override public Token nextToken() {
Token t = super.nextToken();
return t != Token.EOF_TOKEN ? t : MyTokenStream.MY_EOF_TOKEN;
}
}
This is getting too complicated. Just to ensure that all tokens are of
the overridden type, we have to:
- set the TokenLabelType option
- override emit() and nextToken() in the lexer
- override getMissingSymbol() in the parser
- override LT() in the token stream
- upgrade to ANTLR 3.2
and this is all quite fragile and possibly dependent on the Java target.
It clearly calls for a redesign of the way TokenLabelType is handled.
--
David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20091111/d825944c/attachment.bin
More information about the antlr-interest
mailing list