[antlr-interest] multiple lexer

[Ingo Maier]

On Sat, 2004-11-06 at 08:23, Alexey Demakov wrote:
> From: "hst_12345" <hst_12345 at yahoo.com>
> > I am wokring on a project to parse expressions which is part of the 
> > doc.  So I wrote a seperate parser for the expression. And use 
> > multiple lexer. The problem I have is that when switch back, one 
> > token is eaten from the expression parser. Here is my doc syntax:
> > main parser:
> > 
> > EXP { '1.2+3.5' T;  '3.6+6.8' H;}
> > So the main parser rule:
> > expression:
> >      EXP LCURLY 
> >      ( SINGLE_QUOTE {
> >            _selector.select("expressionLex");
> >            expressionParser.Parse();
> >            _selector.select("mainLex");
> >            }
> >        SINGLE_QUOTE character
> >       )+
> >       ;
> > The problem is when switch back, it complains to look for SINGLE_QUOTE 
> > but get T which is next token. I do print out the expression which is 
> > OK that does not include the single quote. So I do not know where 
> > the single quote. why when it gets back to main parser and the single 
> > quote went away.
> 
> It seems to me that expression parser takes the second single quote from
> input strem. In your situation, expression parser needs to see next character
> to decide whether to stop parsing expression. It calls nextToken() removing
> single token from input stream and donesn't consume it desining to stop 
> expression parsing.
> 
> The simplest solution - to have in nested lexer/parser stop condition that doesn't
> use next token. In this case it is single quote at the end of expression.
> And expression lexer will be responsilble for restoring main lexer:
> 
> expr: ... SINGLE_QOUTE { _selector.pop(); }
> 
> So, in main lexer you only push expression lexer:
> 
> expression:
>      EXP LCURLY 
>      ( SINGLE_QUOTE 
>        {
>            _selector.push("expressionLex");
>            expressionParser.Parse();
>            // restored by expression lexer
>            // _selector.select("mainLex");
>        }
>        // it is consumed by expression lexer
>        // SINGLE_QUOTE 
>        character
>       )+
>       ;
> 
> I've never changed lexer from my parser before and have used
> something like this in main lexer:
> 
> SINGLE_QUOTE: '\'' { _selector.push("expressionLex"); }
> 
> So main parser can process tokens from expression lexer:
> 
> expression:
>      EXP LCURLY 
>      ( SINGLE_QUOTE 
>        realExpr // tokens from expression lexer
>        character
>       )+
>       ;
> 
> Regards,
> Alexey

There is no possibility to put the single quote read by the expression
parser/lexer back to the input stream? I mean, so the main parser/lexer
pair could get the chance to parse the single quote by itself (this may
lead to a completely different token for the single quote).

If you want to exchange the expression parser with a different one(s),
you always have to modify the grammar(s) in your solution to fit into
the main parser.

Regards,
Ingo



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/