[antlr-interest] Re: switching lexer while parsing

[Helmut Neukirchen]

> On Tuesday, January 21, 2003, at 10:18 PM, rmsanjivp 
> <rmsanjivp at yahoo.com> wrote:
 >
>>can I switch lexer depending upon the context? I have a problem
>>where in one mode a/b is to be treated as ID DIVIDE ID whereas in
>>another place it must be treated as mere ID. I was thinking or
>>having something that will switch the lexer while parsing depending
>>upon the context.
>>
>>Is it possible?
> 
> Sure.  Easiest way is to use token streams.  See my javadoc example to 
> see how I switch lexers for an embedded language.

I experienced problems, when switching lexers from *within parsers* due to
the lookahead. Switching lexers from within lexers is no problem, but
when you are inside the parser, the wrong lexer might already have
consumed characters and produced wrong tokens. (Worst case: not just
lookead k, but some some syntactic predicate.)

This was already discussed on this list in January 2002...

Since My problem was just academic, I simply stopped that project,
but until then I came to the followingconclusion:

Either move the relevant part which initiates the switch from the parser
into the lexer or implement some checkpointing/unwind mechanisms into
the streams of ANTLR.

Nevertheless, I would be happy to see more convenient solutions!

Best regards,
Helmut
-- 
Helmut Neukirchen                  mailto:neukirchen at itm.uni-luebeck.de
Institute for Telematics                  http://www.itm.uni-luebeck.de
University of Luebeck                           phone: +49 451 500 4867
Ratzeburger Allee 160, D-23538 Luebeck, Germany   fax: +49 451 500 3722


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/