[antlr-interest] TokenRewriteStream usage

[Terence Parr]

hmm...try printing after parsing.  Or, do at least one  
rewrite...perhaps I have a bug that it prints null for now rewrites.
Ter
On Sep 3, 2008, at 5:40 PM, Brisard, Fred D wrote:

> Hi Terence,
>
> Actually I did use rewrite=true and not specified in the grammar.
>
> Here is my test code to drive the lexer and parser --
>
>     public static void main(String args[]) throws Exception
>     {
>         FTP_JOBLexer lex = new FTP_JOBLexer(new ANTLRFileStream("C:\ 
> \testfile.txt"));
>         CommonTokenStream tokens = new CommonTokenStream(lex);
>         int tokensize = tokens.size();
>         String tokenstring = tokens.toString();
>         System.out.println(tokenstring);
>
>         FTP_JOBParser g = new FTP_JOBParser(tokens);
>         try {
>             FTP_JOBParser.program_return parseret = g.program();
>             Object rawTree = parseret.getTree();
>             String [] tokenNames = g.getTokenNames();
>             CommonTree ct = (CommonTree)rawTree;
>             String stringTree = ct.toStringTree();
>             CommonTreePrint pt = new  
> CommonTreePrint(g.getTokenNames());
>             String stringParserTree = pt.print(ct);
>             System.out.println(stringParserTree);
>             String parserstring = g.toString();
>         } catch (RecognitionException e) {
>             e.printStackTrace();
>         }
>     }
>
> The above works OK.
>
> If the only change I make is to replace the CommonTokenStream with  
> TokenRewriteStream in the highlighted line above, I get nothing out  
> of the tokens.toString() and a tokens.size() == 0.
>
> It seemed like that was the logical thing to do since  
> TokenRewriteStream extends CommonTokenStream.
>
> I will debug it further if it’s supposed to work as I’m assuming.  I  
> just didn’t want to be going down the wrong path.
>
> Thanks, Fred
>
> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org 
> ] On Behalf Of Terence Parr
> Sent: Wednesday, September 03, 2008 5:53 PM
> To: Brisard, Fred D
> Cc: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] TokenRewriteStream usage
>
>
> On Sep 3, 2008, at 2:31 PM, Brisard, Fred D wrote:
>
> > I am developing an editor that uses an Antlr-based parser to analyze
> > the source, display it in various forms, allow the user to make
> > modifications and then regenerate the source with those
> > modifications.  From what I read it seems like the
> > TokenRewriteStream is perfect for the job.
> >
> > Right now I have a working parser that operates correctly when using
> > CommonTokenString.  I’m able to parse the source dump an AST based
> > on CommonTree that I built during the parsing.  I figured my first
> > step into rewriting would be to just replace the CommonTokenString
> > object with a TokenRewriteStream object as the result of the lex.  I
> > also added a grammar option of rewrite=true.
> >
>
> that is used for output=template and such.  turn that off for starters
> I think.
> >
> > When I run the lexer I get a null token stream from
> > TokenRewriteStream.toString().
> >
>
> Weird...are you using CommonToken?
>
> Ter
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080903/50dd59c8/attachment.html