[antlr-interest] Fragment tokens are not generated by emit() -- breaks setting TokenLabelType

David-Sarah Hopwood david-sarah at jacaranda.org
Wed Jul 8 12:14:14 PDT 2009


In order to change the token type (so that I can associate another
field with each token), I have overridden emit() in my lexer as shown
below:

grammar Jacaranda;

options {
  TokenLabelType = JacarandaToken;
  language = Java;
}

@lexer::members {
  private String SV = null;

  // See <http://www.antlr.org/wiki/pages/viewpage.action?pageId=1844>.
  public Token emit() {
    JacarandaToken token = new JacarandaToken(input, state.type,
      state.channel, state.tokenStartCharIndex, getCharIndex()-1);
    token.setLine(state.tokenStartLine);
    token.setText(state.text);
    token.setCharPositionInLine(state.tokenStartCharPositionInLine);

    // Transfer the last SV computed by the lexer to the token object.
    token.SV = SV; SV = null;

    emit(token);
    return token;
  }
}

The problem is that the generated code has compilation errors because
not all creation of token objects goes through emit(); there are some
direct uses of 'new CommonToken(...)':

org\jacaranda\verifier\JacarandaLexer.java:4006: incompatible types
found   : org.antlr.runtime.CommonToken
required: org.jacaranda.verifier.JacarandaToken
  d = new CommonToken(input, Token.INVALID_TOKEN_TYPE,
          Token.DEFAULT_CHANNEL, dStart1982, getCharIndex()-1);
      ^

The errors seem to occur in lexer rules where a child rule that is
a fragment is given a name (whether or not that name is used in an
action), for example:

  fragment Foo : d=DecimalDigit ;
  fragment DecimalDigit : '0'..'9' ;

If I remove the option 'TokenLabelType = JacarandaToken', the code
compiles, but this would just be hiding the problem, since I would like
all token objects to be of type JacarandaToken, including those that
are created temporarily for fragments.

As it happens, I can work around this by removing all instances of
named child fragments in my grammar, but I thought I'd report the
bug/limitation here anyway.

-- 
David-Sarah Hopwood  ⚥  http://davidsarah.livejournal.com



More information about the antlr-interest mailing list