[antlr-interest] custom TokenLabelType and EOF/Error tokens

David-Sarah Hopwood david-sarah at jacaranda.org
Tue Nov 10 21:06:10 PST 2009

David-Sarah Hopwood wrote:
> If you are subclassing CommonTokenStream or TokenRewriteStream,
> I think it should be sufficient to override the LT method as follows:
>    protected static MyToken MY_EOF_TOKEN = new MyToken(CharStream.EOF);
>    @Override public Token LT(int k) {
>      Token t = super.LT(k);
>      return t != Token.EOF_TOKEN ? t : MY_EOF_TOKEN;
>    }
> (The EOF_TOKEN doesn't actually exist in the token stream; it is
> returned only when you look ahead using LT.)
> However, I haven't tested this,

I've tested it now; it appears to work.

> and I don't know whether there are any other places where CommonToken is hardcoded.
>> i'm currently using 3.1.3 (since that's what my eclipse ide
>> supports)....  is there is workaround for 3.1.3???
> The workaround I used was:
>  - Delete the TokenLabelType option;
>  - Change my code so that it no longer assumes that all tokens are
>    instances of MyToken. Emitted and error tokens will still be of
>    class MyToken, but fragment and EOF tokens might not be.
>    (Fragment tokens only occur if you refer to a named child fragment
>    in a lexer rule. To check that you're not doing this, search for
>    "new CommonToken" in the generated lexer.)

I forgot to say that this is in addition to overriding getMissingSymbol
as I described earlier.

David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20091111/ef25821c/attachment.bin 

More information about the antlr-interest mailing list