[antlr-interest] Emitting (additional) imaginary tokens in theC target
jimi at temporal-wave.com
Thu Jun 14 07:43:25 PDT 2007
This just reflects the way the java is constructed - you have to follow the entire flow before deciding that emitNew should call emit (if it should then there would only be one function) :-). At the moment it isn't designed for multiple tokens basically.
I think (assuming that you really do need to emit two tokens), that you are looking at the wrong thing - the emit and emitNew already swapped places before the 3.0 snap because of Java changes that I want to stay somewhat in line with).
Having got a token, you want to add the new token into the output stream, hence you can emit, then add the lexer->token to your list/array/whatever then emit another one and add that to your list/array/etc... so the method you probably then want to override is the:
fillBuffer (pANTLR3_COMMON_TOKEN_STREAM tokenStream)
in the token stream, rather than nextToken I think. You can just copy it, and make this code deal with a list/array of tokens. However, then you have to maintain this and so on, whereas at some point I will expand the logic to deal with multiple tokens as it won't be much overhead in the C runtime I think.
As always though, before smashing down the door with a hammer, try the handle. What makes you believe that you need to emit multiple tokens? I would be willing to bet that you don't really need that, though it isn't a given of course.
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Wincent Colaiuta
> Sent: Thursday, June 14, 2007 4:25 AM
> To: Gavin Lambert
> Cc: ANTLR mail-list
> Subject: Re: [antlr-interest] Emitting (additional) imaginary tokens in
> theC target
> El 14/6/2007, a las 12:48, Gavin Lambert escribió:
> > At 22:22 14/06/2007, Wincent Colaiuta wrote:
> > >(Jim, I think you should probably change emit() to
> > >call emitNew() rather than doing "lexer->token =
> > >token;" for this very reason).
> > Actually, I think it'd make more sense for emitNew() to call emit
> > (), since the first constructs a new token then emits it while the
> > second only emits an existing token.
> Actually, emitNew emits an existing token and emit constructs a new
> static void emitNew (pANTLR3_LEXER lexer,
> pANTLR3_COMMON_TOKEN token)
> lexer->token = token; /* Voila! */
> static pANTLR3_COMMON_TOKEN
> emit (pANTLR3_LEXER lexer)
> pANTLR3_COMMON_TOKEN token;
> /* We could check pointers to token factories and so on, but
> * we are in code that we want to run as fast as possible
> * so we are not checking any errors. So make sure you have
> installed an input stream before
> * trying to emit a new token.
> token = lexer->tokFactory->newToken(lexer->tokFactory);
> /* Install the supplied information, and some other bits we
> already know
> * get added automatically, such as the input stream it is
> assoicated with
> * (though it can all be overridden of course)
> token->type = lexer->type;
> token->channel = lexer->channel;
> token->start = lexer->tokenStartCharIndex;
> token->stop = lexer->getCharIndex(lexer) - 1;
> token->line = lexer->tokenStartLine;
> token->charPosition = lexer->tokenStartCharPositionInLine;
> token->text = lexer->text;
> lexer->token = token;
> return token;
More information about the antlr-interest