[antlr-interest] token filtering in C target (antlr 3.0) [repost with correct subject]

[Corno Schraverus]

Thanks Jim for the answer, I found the nextToken method now, I must have
overlooked it.
I think I'm very close to getting it to work but there's one thing that I
don't get.

The tokens that are fed to the parser are identical now (considering 'text'
and 'type') to the ones when using antlr 2.7,
but nonetheless I see that the parser misinterprets the stream for the
tokens I add in between the lexer and the parser.
I think I must be creating the tokens in the wrong way or I must be setting
the text in a wrong way, otherwise, I don't understand what the difference
between the original tokens and my added tokens is.

Is the code below the way it is supposed to be done?
(WhitespaceSignificantTokenStream has a pointer to the lexer)


void setText(pANTLR3_COMMON_TOKEN token, WhitespaceSignificantTokenStream*
wsts, char const* value)
{
  token->setText(token,
wsts->lexer_->pLexer->tokSource->strFactory->newStr(wsts->lexer_->pLexer->to
kSource->strFactory, (pANTLR3_UINT8)value));
}

char* getText(pANTLR3_COMMON_TOKEN token)
{
  return (char*)token->getText(token)->chars;
}

void addToken(WhitespaceSignificantTokenStream* wsts, int type, char* value)
{
  pANTLR3_COMMON_TOKEN tok =
wsts->lexer_->pLexer->tokFactory->newToken(wsts->lexer_->pLexer->tokFactory)
;
  tok->setType(tok, type);
  setText(tok, wsts, value);
  tok->setLine(tok, wsts->lexer_->pLexer->getLine(wsts->lexer_->pLexer));
  wsts->theQueue_.push(tok);
}

TIA,

Corno
 
> nextToken is there. It is a "method" in the tokSource construct, which
> is contained in the lexer. Write your replacement nextToken by copying
> the one in antlr3Lexer.c, then after you create the lexer and add the
> tokenSource, install your own version.
> 
> Jim