[antlr-interest] Fragment tokens are not generated by emit() -- breaks setting TokenLabelType
David-Sarah Hopwood
david-sarah at jacaranda.org
Wed Jul 8 12:14:14 PDT 2009
In order to change the token type (so that I can associate another
field with each token), I have overridden emit() in my lexer as shown
below:
grammar Jacaranda;
options {
TokenLabelType = JacarandaToken;
language = Java;
}
@lexer::members {
private String SV = null;
// See <http://www.antlr.org/wiki/pages/viewpage.action?pageId=1844>.
public Token emit() {
JacarandaToken token = new JacarandaToken(input, state.type,
state.channel, state.tokenStartCharIndex, getCharIndex()-1);
token.setLine(state.tokenStartLine);
token.setText(state.text);
token.setCharPositionInLine(state.tokenStartCharPositionInLine);
// Transfer the last SV computed by the lexer to the token object.
token.SV = SV; SV = null;
emit(token);
return token;
}
}
The problem is that the generated code has compilation errors because
not all creation of token objects goes through emit(); there are some
direct uses of 'new CommonToken(...)':
org\jacaranda\verifier\JacarandaLexer.java:4006: incompatible types
found : org.antlr.runtime.CommonToken
required: org.jacaranda.verifier.JacarandaToken
d = new CommonToken(input, Token.INVALID_TOKEN_TYPE,
Token.DEFAULT_CHANNEL, dStart1982, getCharIndex()-1);
^
The errors seem to occur in lexer rules where a child rule that is
a fragment is given a name (whether or not that name is used in an
action), for example:
fragment Foo : d=DecimalDigit ;
fragment DecimalDigit : '0'..'9' ;
If I remove the option 'TokenLabelType = JacarandaToken', the code
compiles, but this would just be hiding the problem, since I would like
all token objects to be of type JacarandaToken, including those that
are created temporarily for fragments.
As it happens, I can work around this by removing all instances of
named child fragments in my grammar, but I thought I'd report the
bug/limitation here anyway.
--
David-Sarah Hopwood ⚥ http://davidsarah.livejournal.com
More information about the antlr-interest
mailing list