[antlr-interest] C++ TokenStreamSelector

John Reid j.reid at mail.cryst.bbk.ac.uk
Thu Feb 15 03:05:39 PST 2007


Hi,

I'm attaching my parser to a TokenStreamSelector. In some of my parser 
rules I call push on the selector to switch between lexers.

Unfortunately subsequently I do not always get tokens matched from the 
newly pushed lexer. Often I see tokens that have been lexed by the 
previous lexer. These must be buffered somewhere (perhaps in a 
TokenBuffer?).

What is the recommended way to flush this buffer and force re-lexing of 
the input stream? I have seen some documentation referring to 
selector->retry() but this throws an exception that terminates the parse.

The example code always pushes new lexers in the lexer immediately after 
a token has been matched. I would like to push lexers in the parser. 
Perhaps using the selector from the parser is not supported?

Sorry for not posting example code :(

Hope that makes sense and thx in advance for any help,
John.



More information about the antlr-interest mailing list