[antlr-interest] Template Rewrite Rule consumes one more Token than expected

[Felix Dorner]

Gavin Lambert wrote:
> At 10:04 22/04/2008, Felix Dorner wrote:
> >a) The tree parser does not have to re-insert anything in to the
> >token stream, because not hooking a token into the tree does not
> >imply that it's deleted from the input token stream.
>
> No, but the token stream is only input to the parser, not the tree 
> parser.  It's totally irrelevant to the tree parser, as far as I 
> know.  (But maybe you're doing something I don't know about -- see 
> below.)
Yeah, I guess I do something different. Because the excluded tokens are 
not visible to the tree parser, I shouldn't be able to replace them in a 
rewriter that
only sees the AST.
> >grammar Test;
> >options{output=AST;ASTLabelType=CommonTree;}
> >statement : (expression '!'!)* ;
> >expression : 'foo';
> >
> >tree grammar TestTree;
> >options{tokenVocab=Test;output=template;ASTLabelType=CommonTree;
> >rewrite=true;}
> >statement: expression*;
> >expression : 'foo' -> template(text={"bar"}) "<text>";
> >
> >Input "foo!foo!foo!" yields the output "bar!bar!bar!", the '!' are
> >still there... Contrary to the previous example.
>
> Ok, I have no explanation for that, if true.  Unless that's something 
> caused by the 'rewrite=true' option; I'm not familiar with that one.
The "rewrite=true" says that I don't want to create a completely new 
output, but to indicate that I want to barely change _some_ of the 
original tokens in
the stream. If I do this in a  parser, the parser should of course only 
be able to access parts of the original token stream, namely those 
tokens that are hooked into the tree. It's like shooting a bunch of 
bullets through a wall. I can now spray some color at the wall behind, 
but only at where the holes are. My first example also sprayed "around 
the corner", which seems fishy to me.

Felix