[antlr-interest] Fragment tokens are not generated by emit() -- breaks setting TokenLabelType
Jim Idle
jimi at temporal-wave.com
Wed Jul 8 15:46:07 PDT 2009
David-Sarah Hopwood wrote:
> In order to change the token type (so that I can associate another
> field with each token), I have overridden emit() in my lexer as shown
> below:
>
> grammar Jacaranda;
>
> options {
> TokenLabelType = JacarandaToken;
> language = Java;
> }
>
> @lexer::members {
> private String SV = null;
>
> // See <http://www.antlr.org/wiki/pages/viewpage.action?pageId=1844>.
> public Token emit() {
> JacarandaToken token = new JacarandaToken(input, state.type,
> state.channel, state.tokenStartCharIndex, getCharIndex()-1);
> token.setLine(state.tokenStartLine);
> token.setText(state.text);
> token.setCharPositionInLine(state.tokenStartCharPositionInLine);
>
> // Transfer the last SV computed by the lexer to the token object.
> token.SV = SV; SV = null;
>
> emit(token);
> return token;
> }
> }
>
> The problem is that the generated code has compilation errors because
> not all creation of token objects goes through emit(); there are some
> direct uses of 'new CommonToken(...)':
>
> org\jacaranda\verifier\JacarandaLexer.java:4006: incompatible types
> found : org.antlr.runtime.CommonToken
> required: org.jacaranda.verifier.JacarandaToken
> d = new CommonToken(input, Token.INVALID_TOKEN_TYPE,
> Token.DEFAULT_CHANNEL, dStart1982, getCharIndex()-1);
> ^
>
Fixed (in Java target so far) by: http://www.antlr.org/jira/browse/ANTLR-398
Jim
More information about the antlr-interest
mailing list