[antlr-interest] Emitting (additional) imaginary tokens in the C target

[Wincent Colaiuta]

El 14/6/2007, a las 12:48, Gavin Lambert escribió:

> At 22:22 14/06/2007, Wincent Colaiuta wrote:
> >(Jim, I think you should probably change emit() to
> >call emitNew() rather than doing "lexer->token =
> >token;" for this very reason).
>
> Actually, I think it'd make more sense for emitNew() to call emit 
> (), since the first constructs a new token then emits it while the  
> second only emits an existing token.

Actually, emitNew emits an existing token and emit constructs a new  
token:

static void emitNew         (pANTLR3_LEXER lexer,   
pANTLR3_COMMON_TOKEN token)
{
     lexer->token    = token;    /* Voila!   */
}

static pANTLR3_COMMON_TOKEN
emit        (pANTLR3_LEXER lexer)
{
     pANTLR3_COMMON_TOKEN        token;

     /* We could check pointers to token factories and so on, but
      * we are in code that we want to run as fast as possible
      * so we are not checking any errors. So make sure you have  
installed an input stream before
      * trying to emit a new token.
      */
     token   = lexer->tokFactory->newToken(lexer->tokFactory);

     /* Install the supplied information, and some other bits we  
already know
      * get added automatically, such as the input stream it is  
assoicated with
      * (though it can all be overridden of course)
      */
     token->type             = lexer->type;
     token->channel          = lexer->channel;
     token->start            = lexer->tokenStartCharIndex;
     token->stop             = lexer->getCharIndex(lexer) - 1;
     token->line             = lexer->tokenStartLine;
     token->charPosition     = lexer->tokenStartCharPositionInLine;
     token->text             = lexer->text;

     lexer->token            = token;

     return  token;
}

Cheers,
Wincent