[antlr-interest] C++ TokenStreamSelector

[Ric Klaren]

Hi,

On 2/15/07, John Reid <j.reid at mail.cryst.bbk.ac.uk> wrote:
> I'm attaching my parser to a TokenStreamSelector. In some of my parser
> rules I call push on the selector to switch between lexers.

This does not work in the general case (due to lookahead,
tokenbuffering and presence of syntactic predicates).

> What is the recommended way to flush this buffer and force re-lexing of
> the input stream?

There is no such mechanism. You might get something to work with very
creative use of mark, rewind on the buffer and adding code to
invalidate/reset the state of the lookahead. But this requires a
*very* *very* good understanding of your parser and how it parses.
E.g. you have to mark the input at the start of a rule if you suspect
that a switch might be necessary and rewind and cleanup if it fails.
Or unregister the mark if it was not needed (e.g. no switch needed)
(in short: a maintenance nightmare)

I would not go tread way unless I *really* had no other option. E.g.
more passes, uses AST's.. maybe use tokenstream rewriting. It depends
on what you want to accomplish.

Cheers,

Ric