[antlr-interest] Template Rewrite Rule consumes one more Token than expected

Felix Dorner felix_do at web.de
Tue Apr 22 03:53:01 PDT 2008

Gavin Lambert wrote:
> At 10:04 22/04/2008, Felix Dorner wrote:
> >a) The tree parser does not have to re-insert anything in to the
> >token stream, because not hooking a token into the tree does not
> >imply that it's deleted from the input token stream.
> No, but the token stream is only input to the parser, not the tree 
> parser.  It's totally irrelevant to the tree parser, as far as I 
> know.  (But maybe you're doing something I don't know about -- see 
> below.)
Yeah, I guess I do something different. Because the excluded tokens are 
not visible to the tree parser, I shouldn't be able to replace them in a 
rewriter that
only sees the AST.
> >grammar Test;
> >options{output=AST;ASTLabelType=CommonTree;}
> >statement : (expression '!'!)* ;
> >expression : 'foo';
> >
> >tree grammar TestTree;
> >options{tokenVocab=Test;output=template;ASTLabelType=CommonTree;
> >rewrite=true;}
> >statement: expression*;
> >expression : 'foo' -> template(text={"bar"}) "<text>";
> >
> >Input "foo!foo!foo!" yields the output "bar!bar!bar!", the '!' are
> >still there... Contrary to the previous example.
> Ok, I have no explanation for that, if true.  Unless that's something 
> caused by the 'rewrite=true' option; I'm not familiar with that one.
The "rewrite=true" says that I don't want to create a completely new 
output, but to indicate that I want to barely change _some_ of the 
original tokens in
the stream. If I do this in a  parser, the parser should of course only 
be able to access parts of the original token stream, namely those 
tokens that are hooked into the tree. It's like shooting a bunch of 
bullets through a wall. I can now spray some color at the wall behind, 
but only at where the holes are. My first example also sprayed "around 
the corner", which seems fishy to me.


More information about the antlr-interest mailing list