[antlr-interest] ANTLR3 Nested parser

Thomas Brandon tbrandonau at gmail.com
Tue Jan 22 01:27:20 PST 2008


Check out the island-grammar example. It shows lexer based nesting. The
pertinent code in the main lexer is:
JAVADOC : '/**'
          {
            // create a new javadoc lexer/parser duo that feeds
            // off the current input stream
            System.out.println("enter javadoc");
            JavadocLexer j = new JavadocLexer(input);
            CommonTokenStream tokens = new CommonTokenStream(j);
            tokens.discardTokenType(JavadocLexer.WS);
            JavadocParser p = new JavadocParser(tokens);
            p.comment();
            // returns a JAVADOC token to the java parser but on a
            // different channel than the normal token stream so it
            // doesn't get in the way.
            $channel = JAVADOC_CHANNEL;
          }
        ;
And in the nested lexer:
/** When the javadoc parser sees end-of-comment it just says 'I'm done',
which
 *  consumes the tokens and forces this javadoc parser (feeding
 *  off the input stream currently) to exit.  It returns from
 *  method comment(), which was called from JAVADOC action in the
 *  Simple parser's lexer.
 */
END     : '*/' {token = Token.EOF_TOKEN;}
          {System.out.println("exit javadoc");}
        ;

Your code in the outer lexer looks OK, just passing the same input stream
should keep it synched. Check your code for exiting the inner lexer.

Tom.
On Jan 22, 2008 8:00 PM, Bertalan Fodor (LilyPondTool) <
lilypondtool at organum.hu> wrote:

>  Thank you for your answer.
> If I understand your suggestion correctly, the problem with it is that I
> can't lex the symbol: actually it is an embedded language, so only the
> nested lexer and parser can find the end of the embedded part. So actually I
> would need the following: feed the nested lexer/parser with all the input
> beginning from the '/**' symbol and let it parse it. Then when the parsing
> is over, I'd like to set the non-nested lexer/parser to the end position of
> the nested parsing. So the problem is how to set the input stream position
> to the end of the embedded part, either rewinding or forward winding. Maybe
> I can use the return value of the nested rootRule() to find the position.
> However I could not find a way yet how to do this all.
>
> Anyway, if you have some example of some nested parsing, that could
> probably effectively help me.
>
> Thanks,
>
> Bert
>
>
> Harald M. Müller wrote:
>
> I wouldn't do it like this.
> If you want to really do this in the (non-nested) lexer: "Lex" the symbol;
> and then start a NEW StringReader on the symbols's getText(), from which you
> feed your (nested) lexer and parser.
> Maybe you want to do this in the (non-nested) parser ... easy if the symbol
> turns up there; a little work if you pushed the symbol into the HIDDEN
> channel (or some other channel) in the (non-nested) lexer, so that you have
> to "undig" it somehow in the (non-nested) parser
> Hope this very short explanation helps.
> Regards
> Harald
>
>
>
>  -----Original Message-----
> From: antlr-interest-bounces at antlr.org
> [mailto:antlr-interest-bounces at antlr.org <antlr-interest-bounces at antlr.org>] On Behalf Of Bertalan Fodor
> Sent: Tuesday, January 22, 2008 12:45 AM
> To: antlr-interest at antlr.org
> Subject: [antlr-interest] ANTLR3 Nested parser
>
> Hi,
>
> I'm creating a parser with nested parser. To make it simple I
> tell you my problem as if I was doing Javadoc parsing inside
> a Java parser.
> So I have this in my lexer:
> JAVADOC: '/**' { JavadocParser javadocParser = new
> JavadocParser(new CommonTokenStream(new
> JavadocLexer(this.input))); javadocParser.rootRule(); }
>
> The problem is that while this code correctly switches to the
> Javadoc lexer, and parses the Javadoc parts correctly, upon
> returning from the Javadoc parsing, the character stream is
> not correctly positioned.
>
> Can you help me how to achieve the nested parsing in antlr 3?
>
> Thank you very much,
>
> Bertalan Fodor
>
> ps I've already asked this on this list, but got no answer,
> so I tried to make my question more simple.
>
>
>
>
>
> --
> LilyPondTool is the editor for LilyPond files.
> See http://lilypondtool.organum.hu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080122/a5e06450/attachment.html 


More information about the antlr-interest mailing list