[antlr-interest] multiple lexer

[Alexey Demakov]

From: "hst_12345" <hst_12345 at yahoo.com>
> I am wokring on a project to parse expressions which is part of the 
> doc.  So I wrote a seperate parser for the expression. And use 
> multiple lexer. The problem I have is that when switch back, one 
> token is eaten from the expression parser. Here is my doc syntax:
> main parser:
> 
> EXP { '1.2+3.5' T;  '3.6+6.8' H;}
> So the main parser rule:
> expression:
>      EXP LCURLY 
>      ( SINGLE_QUOTE {
>            _selector.select("expressionLex");
>            expressionParser.Parse();
>            _selector.select("mainLex");
>            }
>        SINGLE_QUOTE character
>       )+
>       ;
> The problem is when switch back, it complains to look for SINGLE_QUOTE 
> but get T which is next token. I do print out the expression which is 
> OK that does not include the single quote. So I do not know where 
> the single quote. why when it gets back to main parser and the single 
> quote went away.

It seems to me that expression parser takes the second single quote from
input strem. In your situation, expression parser needs to see next character
to decide whether to stop parsing expression. It calls nextToken() removing
single token from input stream and donesn't consume it desining to stop 
expression parsing.

The simplest solution - to have in nested lexer/parser stop condition that doesn't
use next token. In this case it is single quote at the end of expression.
And expression lexer will be responsilble for restoring main lexer:

expr: ... SINGLE_QOUTE { _selector.pop(); }

So, in main lexer you only push expression lexer:

expression:
     EXP LCURLY 
     ( SINGLE_QUOTE 
       {
           _selector.push("expressionLex");
           expressionParser.Parse();
           // restored by expression lexer
           // _selector.select("mainLex");
       }
       // it is consumed by expression lexer
       // SINGLE_QUOTE 
       character
      )+
      ;

I've never changed lexer from my parser before and have used
something like this in main lexer:

SINGLE_QUOTE: '\'' { _selector.push("expressionLex"); }

So main parser can process tokens from expression lexer:

expression:
     EXP LCURLY 
     ( SINGLE_QUOTE 
       realExpr // tokens from expression lexer
       character
      )+
      ;

Regards,
Alexey

-----
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com




 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/