[antlr-interest] Template Rewrite Rule consumes one more Token than expected

Gavin Lambert antlr at mirality.co.nz
Tue Apr 22 03:17:21 PDT 2008


At 10:04 22/04/2008, Felix Dorner wrote:
 >a) The tree parser does not have to re-insert anything in to the 

 >token stream, because not hooking a token into the tree does not 

 >imply that it's deleted from the input token stream.

No, but the token stream is only input to the parser, not the tree 
parser.  It's totally irrelevant to the tree parser, as far as I 
know.  (But maybe you're doing something I don't know about -- see 
below.)

 >b) In the book, there's an example where Terence rewrites a 
token
 >stream with a tree grammar, he  excludes semicolons from the 
AST,
 >but in the rewritten token stream, the tokens are still there
 >( I admit that I am not sure if he does some magic to make the
 >semicolons appear..)

I don't have the book, so I can't speak to that.

 >grammar Test;
 >options{output=AST;ASTLabelType=CommonTree;}
 >statement : (expression '!'!)* ;
 >expression : 'foo';
 >
 >tree grammar TestTree;
 >options{tokenVocab=Test;output=template;ASTLabelType=CommonTree;
 >rewrite=true;}
 >statement: expression*;
 >expression : 'foo' -> template(text={"bar"}) "<text>";
 >
 >Input "foo!foo!foo!" yields the output "bar!bar!bar!", the '!' 
are
 >still there... Contrary to the previous example.

Ok, I have no explanation for that, if true.  Unless that's 
something caused by the 'rewrite=true' option; I'm not familiar 
with that one.



More information about the antlr-interest mailing list