[antlr-interest] Multiplexing question - no lexer token to switch, context necessary

Monty Zukowski monty at codetransform.com
Tue Sep 7 08:34:27 PDT 2004

On Sep 7, 2004, at 5:15 AM, Maximilian Störzer wrote:

> Hi everybody!
> I am stuck with a multiplexing problem. I understand that in general 
> it is
> no good idea to switch the lexer from the parser.
> However I have the problem that I would need a different lexer 
> dependent on
> the context. There is *no* token (like e.g. "/**" for JavaDoc) to allow
> switching the lexer from within the lexer. Only the parser knows about 
> this
> necessary context.
> The problem is that - depending on context - foo*bar should be parsed 
> as
> IDENT STAR IDENT or as WILDCARD_IDENT, but, again, there is no single 
> token
> to indicate this state change, only the context (i.e. the current 
> parser
> rule the parser sits in).

So do it with a parser rule--
wildcard_ident: IDENT STAR IDENT;

You probably don't like that because then you could recognize foo * bar 
as well, assuming you are skipping whitespace like most parsers.

Alternatively see if you can use a filter.  Read my article 
http://www.codetransform.com/filterexample.html.  I really have to edit 
it to be more to the point, but the gist is to have a filter that is 
actually a parser and can start parsing a particular rule if it can, 
otherwise pass the token through.  So you would not skip whitespace in 
the lexer, instead you would have a filter to do that.  The filter 
would also have the rule that required WILDCARD_IDENT so as to change 

You may also be able to have enough context introduced into your lexer 
to recognize it.  But without any more specifics it is hard to help.


ANTLR & Java Consultant -- http://www.codetransform.com
ANSI C/GCC transformation toolkit -- 
Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html

Yahoo! Groups Links

<*> To visit your group on the web, go to:

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:

More information about the antlr-interest mailing list