[antlr-interest] antlr-interest Digest, Vol 37, Issue 54
antlr-interest at schraverus.com
Fri Dec 28 14:08:20 PST 2007
Thanks Jim for the answer, I found the nextToken method now, I must have
I think I'm very close to getting it to work but there's one thing that I
The tokens that are fed to the parser are identical now (considering 'text'
and 'type') to the ones when using antlr 2.7,
but nonetheless I see that the parser misinterprets the stream for the
tokens I add in between the lexer and the parser.
I think I must be creating the tokens in the wrong way or I must be setting
the text in a wrong way, otherwise, I don't understand what the difference
between the original tokens and my added tokens is.
Is the code below the way it is supposed to be done?
(WhitespaceSignificantTokenStream has a pointer to the lexer)
void setText(pANTLR3_COMMON_TOKEN token, WhitespaceSignificantTokenStream*
wsts, char const* value)
char* getText(pANTLR3_COMMON_TOKEN token)
void addToken(WhitespaceSignificantTokenStream* wsts, int type, char* value)
pANTLR3_COMMON_TOKEN tok =
setText(tok, wsts, value);
> nextToken is there. It is a "method" in the tokSource construct, which
> is contained in the lexer. Write your replacement nextToken by copying
> the one in antlr3Lexer.c, then after you create the lexer and add the
> tokenSource, install your own version.
> > -----Original Message-----
> > From: Corno Schraverus [mailto:antlr-interest at schraverus.com]
> > Sent: Wednesday, December 19, 2007 5:01 PM
> > To: antlr-interest at antlr.org
> > Subject: [antlr-interest] token filtering in C target (antlr 3.0)
> > Hi all,
> > I'm trying to port a whitespace significant grammar from 2.7 to 3.0.
> > (It is almost identical to the python parser available for 2.7;
> > all that has to be done is override the 'nextToken' method.)
> > I'm going through the sources of the runtime library for the C target
> > cannot find the nextToken method(function) and if I'm correct it's
> > with
> > _LA(). However _LA() is more complex: arbitrary lookahead or
> > I
> > have to implement this functionality to get it working correctly?
> > greetings,
> > Corno
More information about the antlr-interest