[antlr-interest] antlr-interest Digest, Vol 37, Issue 54

[Corno Schraverus]

Thanks Jim for the answer, I found the nextToken method now, I must have
overlooked it.
I think I'm very close to getting it to work but there's one thing that I
don't get.

The tokens that are fed to the parser are identical now (considering 'text'
and 'type') to the ones when using antlr 2.7,
but nonetheless I see that the parser misinterprets the stream for the
tokens I add in between the lexer and the parser.
I think I must be creating the tokens in the wrong way or I must be setting
the text in a wrong way, otherwise, I don't understand what the difference
between the original tokens and my added tokens is.

Is the code below the way it is supposed to be done?
(WhitespaceSignificantTokenStream has a pointer to the lexer)


void setText(pANTLR3_COMMON_TOKEN token, WhitespaceSignificantTokenStream*
wsts, char const* value)
{
  token->setText(token,
wsts->lexer_->pLexer->tokSource->strFactory->newStr(wsts->lexer_->pLexer->to
kSource->strFactory, (pANTLR3_UINT8)value));
}

char* getText(pANTLR3_COMMON_TOKEN token)
{
  return (char*)token->getText(token)->chars;
}

void addToken(WhitespaceSignificantTokenStream* wsts, int type, char* value)
{
  pANTLR3_COMMON_TOKEN tok =
wsts->lexer_->pLexer->tokFactory->newToken(wsts->lexer_->pLexer->tokFactory)
;
  tok->setType(tok, type);
  setText(tok, wsts, value);
  tok->setLine(tok, wsts->lexer_->pLexer->getLine(wsts->lexer_->pLexer));
  wsts->theQueue_.push(tok);
}

TIA,

Corno
 
> nextToken is there. It is a "method" in the tokSource construct, which
> is contained in the lexer. Write your replacement nextToken by copying
> the one in antlr3Lexer.c, then after you create the lexer and add the
> tokenSource, install your own version.
> 
> Jim
> 
> > -----Original Message-----
> > From: Corno Schraverus [mailto:antlr-interest at schraverus.com]
> > Sent: Wednesday, December 19, 2007 5:01 PM
> > To: antlr-interest at antlr.org
> > Subject: [antlr-interest] token filtering in C target (antlr 3.0)
> >
> > Hi all,
> >
> > I'm trying to port a whitespace significant grammar from 2.7 to 3.0.
> > (It is almost identical to the python parser available for 2.7;
> basically
> > all that has to be done is override the 'nextToken' method.)
> > I'm going through the sources of the runtime library for the C target
> but
> > cannot find the nextToken method(function) and if I'm correct it's
> done
> > with
> > _LA(). However _LA() is more complex: arbitrary lookahead or
> lookback.
> Do
> > I
> > have to implement this functionality to get it working correctly?
> >
> > greetings,
> >
> > Corno
> >