[antlr-interest] Emitting (additional) imaginary tokens in theC target

Jim Idle jimi at temporal-wave.com
Thu Jun 14 07:43:25 PDT 2007

This just reflects the way the java is constructed - you have to follow the entire flow before deciding that emitNew should call emit (if it should then there would only be one function) :-). At the moment it isn't designed for multiple tokens basically.
I think (assuming that you really do need to emit two tokens), that you are looking at the wrong thing - the emit and emitNew already swapped places before the 3.0 snap because of Java changes that I want to stay somewhat in line with). 

Having got a token, you want to add the new token into the output stream, hence you can emit, then add the lexer->token to your list/array/whatever then emit another one and add that to your list/array/etc... so the method you probably then want to override is the:

static void
fillBuffer  (pANTLR3_COMMON_TOKEN_STREAM tokenStream)

in the token stream, rather than nextToken I think. You can just copy it, and make this code deal with a list/array of tokens. However, then you have to maintain this and so on, whereas at some point I will expand the logic to deal with multiple tokens as it won't be much overhead in the C runtime I think.

As always though, before smashing down the door with a hammer, try the handle. What makes you believe that you need to emit multiple tokens? I would be willing to bet that you don't really need that, though it isn't a given of course.


> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Wincent Colaiuta
> Sent: Thursday, June 14, 2007 4:25 AM
> To: Gavin Lambert
> Cc: ANTLR mail-list
> Subject: Re: [antlr-interest] Emitting (additional) imaginary tokens in
> theC target
> El 14/6/2007, a las 12:48, Gavin Lambert escribió:
> > At 22:22 14/06/2007, Wincent Colaiuta wrote:
> > >(Jim, I think you should probably change emit() to
> > >call emitNew() rather than doing "lexer->token =
> > >token;" for this very reason).
> >
> > Actually, I think it'd make more sense for emitNew() to call emit
> > (), since the first constructs a new token then emits it while the
> > second only emits an existing token.
> Actually, emitNew emits an existing token and emit constructs a new
> token:
> static void emitNew         (pANTLR3_LEXER lexer,
> {
>      lexer->token    = token;    /* Voila!   */
> }
> emit        (pANTLR3_LEXER lexer)
> {
>      pANTLR3_COMMON_TOKEN        token;
>      /* We could check pointers to token factories and so on, but
>       * we are in code that we want to run as fast as possible
>       * so we are not checking any errors. So make sure you have
> installed an input stream before
>       * trying to emit a new token.
>       */
>      token   = lexer->tokFactory->newToken(lexer->tokFactory);
>      /* Install the supplied information, and some other bits we
> already know
>       * get added automatically, such as the input stream it is
> assoicated with
>       * (though it can all be overridden of course)
>       */
>      token->type             = lexer->type;
>      token->channel          = lexer->channel;
>      token->start            = lexer->tokenStartCharIndex;
>      token->stop             = lexer->getCharIndex(lexer) - 1;
>      token->line             = lexer->tokenStartLine;
>      token->charPosition     = lexer->tokenStartCharPositionInLine;
>      token->text             = lexer->text;
>      lexer->token            = token;
>      return  token;
> }
> Cheers,
> Wincent

More information about the antlr-interest mailing list