[antlr-interest] Multiplexing question - no lexer token to switch, context necessary

[Monty Zukowski]

On Sep 7, 2004, at 5:15 AM, Maximilian Störzer wrote:

> Hi everybody!
>
> I am stuck with a multiplexing problem. I understand that in general 
> it is
> no good idea to switch the lexer from the parser.
>
> However I have the problem that I would need a different lexer 
> dependent on
> the context. There is *no* token (like e.g. "/**" for JavaDoc) to allow
> switching the lexer from within the lexer. Only the parser knows about 
> this
> necessary context.
>
> The problem is that - depending on context - foo*bar should be parsed 
> as
> IDENT STAR IDENT or as WILDCARD_IDENT, but, again, there is no single 
> token
> to indicate this state change, only the context (i.e. the current 
> parser
> rule the parser sits in).

So do it with a parser rule--
wildcard_ident: IDENT STAR IDENT;

You probably don't like that because then you could recognize foo * bar 
as well, assuming you are skipping whitespace like most parsers.

Alternatively see if you can use a filter.  Read my article 
http://www.codetransform.com/filterexample.html.  I really have to edit 
it to be more to the point, but the gist is to have a filter that is 
actually a parser and can start parsing a particular rule if it can, 
otherwise pass the token through.  So you would not skip whitespace in 
the lexer, instead you would have a filter to do that.  The filter 
would also have the rule that required WILDCARD_IDENT so as to change 
IDENT STAR IDENT to WILDCARD_IDENT.

You may also be able to have enough context introduced into your lexer 
to recognize it.  But without any more specifics it is hard to help.

Monty

ANTLR & Java Consultant -- http://www.codetransform.com
ANSI C/GCC transformation toolkit -- 
http://www.codetransform.com/gcc.html
Embrace the Decay -- http://www.codetransform.com/EmbraceDecay.html



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/