[antlr-interest] TokenRewriteStream usage

Brisard, Fred D Fred.Brisard at ca.com
Wed Sep 3 17:40:57 PDT 2008


Hi Terence,

 

Actually I did use rewrite=true and not specified in the grammar.

 

Here is my test code to drive the lexer and parser --

 

    public static void main(String args[]) throws Exception

    {

        FTP_JOBLexer lex = new FTP_JOBLexer(new
ANTLRFileStream("C:\\testfile.txt"));

        CommonTokenStream tokens = new CommonTokenStream(lex);

        int tokensize = tokens.size();

        String tokenstring = tokens.toString();

        System.out.println(tokenstring);

                

        FTP_JOBParser g = new FTP_JOBParser(tokens);

        try {

            FTP_JOBParser.program_return parseret = g.program();

            Object rawTree = parseret.getTree();

            String [] tokenNames = g.getTokenNames();

            CommonTree ct = (CommonTree)rawTree;

            String stringTree = ct.toStringTree();

            CommonTreePrint pt = new CommonTreePrint(g.getTokenNames());

            String stringParserTree = pt.print(ct);

            System.out.println(stringParserTree);

            String parserstring = g.toString();

        } catch (RecognitionException e) {

            e.printStackTrace();

        }

    }

 

The above works OK.  

 

If the only change I make is to replace the CommonTokenStream with
TokenRewriteStream in the highlighted line above, I get nothing out of
the tokens.toString() and a tokens.size() == 0.

 

It seemed like that was the logical thing to do since TokenRewriteStream
extends CommonTokenStream.  

 

I will debug it further if it's supposed to work as I'm assuming.  I
just didn't want to be going down the wrong path.

 

Thanks, Fred

 

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Terence Parr
Sent: Wednesday, September 03, 2008 5:53 PM
To: Brisard, Fred D
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] TokenRewriteStream usage

 

 

On Sep 3, 2008, at 2:31 PM, Brisard, Fred D wrote:

 

> I am developing an editor that uses an Antlr-based parser to analyze  

> the source, display it in various forms, allow the user to make  

> modifications and then regenerate the source with those  

> modifications.  From what I read it seems like the  

> TokenRewriteStream is perfect for the job.

> 

> Right now I have a working parser that operates correctly when using  

> CommonTokenString.  I'm able to parse the source dump an AST based  

> on CommonTree that I built during the parsing.  I figured my first  

> step into rewriting would be to just replace the CommonTokenString  

> object with a TokenRewriteStream object as the result of the lex.  I  

> also added a grammar option of rewrite=true.

> 

 

that is used for output=template and such.  turn that off for starters  

I think.

> 

> When I run the lexer I get a null token stream from  

> TokenRewriteStream.toString().

> 

 

Weird...are you using CommonToken?

 

Ter

 

List: http://www.antlr.org/mailman/listinfo/antlr-interest

Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-addr
ess

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080903/f117ef00/attachment.html 


More information about the antlr-interest mailing list