[antlr-interest] TokenRewriteStream usage
Brisard, Fred D
Fred.Brisard at ca.com
Wed Sep 3 17:40:57 PDT 2008
Hi Terence,
Actually I did use rewrite=true and not specified in the grammar.
Here is my test code to drive the lexer and parser --
public static void main(String args[]) throws Exception
{
FTP_JOBLexer lex = new FTP_JOBLexer(new
ANTLRFileStream("C:\\testfile.txt"));
CommonTokenStream tokens = new CommonTokenStream(lex);
int tokensize = tokens.size();
String tokenstring = tokens.toString();
System.out.println(tokenstring);
FTP_JOBParser g = new FTP_JOBParser(tokens);
try {
FTP_JOBParser.program_return parseret = g.program();
Object rawTree = parseret.getTree();
String [] tokenNames = g.getTokenNames();
CommonTree ct = (CommonTree)rawTree;
String stringTree = ct.toStringTree();
CommonTreePrint pt = new CommonTreePrint(g.getTokenNames());
String stringParserTree = pt.print(ct);
System.out.println(stringParserTree);
String parserstring = g.toString();
} catch (RecognitionException e) {
e.printStackTrace();
}
}
The above works OK.
If the only change I make is to replace the CommonTokenStream with
TokenRewriteStream in the highlighted line above, I get nothing out of
the tokens.toString() and a tokens.size() == 0.
It seemed like that was the logical thing to do since TokenRewriteStream
extends CommonTokenStream.
I will debug it further if it's supposed to work as I'm assuming. I
just didn't want to be going down the wrong path.
Thanks, Fred
-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Terence Parr
Sent: Wednesday, September 03, 2008 5:53 PM
To: Brisard, Fred D
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] TokenRewriteStream usage
On Sep 3, 2008, at 2:31 PM, Brisard, Fred D wrote:
> I am developing an editor that uses an Antlr-based parser to analyze
> the source, display it in various forms, allow the user to make
> modifications and then regenerate the source with those
> modifications. From what I read it seems like the
> TokenRewriteStream is perfect for the job.
>
> Right now I have a working parser that operates correctly when using
> CommonTokenString. I'm able to parse the source dump an AST based
> on CommonTree that I built during the parsing. I figured my first
> step into rewriting would be to just replace the CommonTokenString
> object with a TokenRewriteStream object as the result of the lex. I
> also added a grammar option of rewrite=true.
>
that is used for output=template and such. turn that off for starters
I think.
>
> When I run the lexer I get a null token stream from
> TokenRewriteStream.toString().
>
Weird...are you using CommonToken?
Ter
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-addr
ess
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080903/f117ef00/attachment.html
More information about the antlr-interest
mailing list