[antlr-interest] Fragment tokens are not generated by emit() -- breaks setting TokenLabelType

[Jim Idle]

David-Sarah Hopwood wrote:
> In order to change the token type (so that I can associate another
> field with each token), I have overridden emit() in my lexer as shown
> below:
>
> grammar Jacaranda;
>
> options {
>   TokenLabelType = JacarandaToken;
>   language = Java;
> }
>
> @lexer::members {
>   private String SV = null;
>
>   // See <http://www.antlr.org/wiki/pages/viewpage.action?pageId=1844>.
>   public Token emit() {
>     JacarandaToken token = new JacarandaToken(input, state.type,
>       state.channel, state.tokenStartCharIndex, getCharIndex()-1);
>     token.setLine(state.tokenStartLine);
>     token.setText(state.text);
>     token.setCharPositionInLine(state.tokenStartCharPositionInLine);
>
>     // Transfer the last SV computed by the lexer to the token object.
>     token.SV = SV; SV = null;
>
>     emit(token);
>     return token;
>   }
> }
>
> The problem is that the generated code has compilation errors because
> not all creation of token objects goes through emit(); there are some
> direct uses of 'new CommonToken(...)':
>
> org\jacaranda\verifier\JacarandaLexer.java:4006: incompatible types
> found   : org.antlr.runtime.CommonToken
> required: org.jacaranda.verifier.JacarandaToken
>   d = new CommonToken(input, Token.INVALID_TOKEN_TYPE,
>           Token.DEFAULT_CHANNEL, dStart1982, getCharIndex()-1);
>       ^
>   
Fixed (in Java target so far) by: http://www.antlr.org/jira/browse/ANTLR-398

Jim