[antlr-interest] C Target - Where is it safe to switch input streams?

[A Z]

Hello all,

  In the example C grammar, PUSHSTREAM() is called in the middle of a lexer
rule to implement the include preprocessor directive. I followed this
approach and everything works fine except the current token is incomplete.
The channel and USER values get set but the text is blank. All the tokens
from the included file are correct.

My  __LINE__ directives work the same way as includes and look something
like this:

DIR_LINE :
  '`__LINE__'
    {ctx->pLexerData->dirLineText();runNewBuffer(ctx);$channel=HIDDEN;};

static void runNewBuffer(pSVLexer ctx)
{
    pANTLR3_INPUT_STREAM input =
antlr3NewAsciiStringInPlaceStream(newBufferData,newBufferSize,newBufferName);
    if(input == NULL)
       ANTLR3_FPRINTF(stderr, "Unable to open buffer \%s due to malloc()
failure1\n",newBufferName);

    PUSHSTREAM(input);

    ctx->pLexerData->inputList.push(input); //Mirror what ANTLR does
}

I found if I remove runNewBuffer() then the DIR_LINE token correctly gets
it's text set to "`__LINE__", otherwise the text is blank but the channel is
set correctly in either case. While trying to fix this I found it crashes if
PUSHSTREAM is placed after emit() inside nextTokenStr(). If I move it to the
nextToken() function, it seems to work as intended. Are there any side
effects to switching the input streams inside nextToken()? Is there another
way to run an action after a certain token has been completed?

Thanks