[antlr-interest] custom TokenLabelType and EOF/Error tokens
David-Sarah Hopwood
david-sarah at jacaranda.org
Tue Nov 10 21:06:10 PST 2009
David-Sarah Hopwood wrote:
> If you are subclassing CommonTokenStream or TokenRewriteStream,
> I think it should be sufficient to override the LT method as follows:
>
> protected static MyToken MY_EOF_TOKEN = new MyToken(CharStream.EOF);
>
> @Override public Token LT(int k) {
> Token t = super.LT(k);
> return t != Token.EOF_TOKEN ? t : MY_EOF_TOKEN;
> }
>
> (The EOF_TOKEN doesn't actually exist in the token stream; it is
> returned only when you look ahead using LT.)
>
> However, I haven't tested this,
I've tested it now; it appears to work.
> and I don't know whether there are any other places where CommonToken is hardcoded.
>
>> i'm currently using 3.1.3 (since that's what my eclipse ide
>> supports).... is there is workaround for 3.1.3???
>
> The workaround I used was:
>
> - Delete the TokenLabelType option;
>
> - Change my code so that it no longer assumes that all tokens are
> instances of MyToken. Emitted and error tokens will still be of
> class MyToken, but fragment and EOF tokens might not be.
> (Fragment tokens only occur if you refer to a named child fragment
> in a lexer rule. To check that you're not doing this, search for
> "new CommonToken" in the generated lexer.)
I forgot to say that this is in addition to overriding getMissingSymbol
as I described earlier.
--
David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 292 bytes
Desc: OpenPGP digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20091111/ef25821c/attachment.bin
More information about the antlr-interest
mailing list