[antlr-interest] Duplicating and modifying portions of the input

Geraint North geraint at transitive.com
Thu Jun 12 04:41:27 PDT 2008


Thanks for the suggestion - I've now got something that works well.

What I've done, as I've traversed the AST of the template declaration  
to replace the template parameters with an actual value, is to build  
up a map of "don't print that token, print this CommonTree instead".   
When I want to print out the "instantiated" template, I can iterate  
over the Token Stream from getTokenStartIndex() to  
getTokenStopIndex(), printing out each Token, unless it appears in my  
map, in which case I print out the tokens from the  
getTokenStartIndex() to the getTokenStopIndex() of the tree that the  
placeholder was replaced with.

This has precisely the effect that I was after - printing out the  
template verbatim, with all whitespace and comments preserved but with  
the substitutions made.

Thanks,
Geraint.

On 10 Jun 2008, at 22:00, Gavin Lambert wrote:

> At 04:11 11/06/2008, Geraint North wrote:
> >It seems that if the simple deep copy of the AST that I am
> >performing would also preserve the hidden tokens (does it?),
> >then I should just be able to feed to resulting expanded tree
> >into a "null" rewriter, and that should produce the expanded
> >test, comments and all - is that correct, or do I have to do
> >something to ensure that the token stream is also deep copied?
>
> Hidden tokens will usually not make it into the AST in the first  
> place, so copying the tree won't help you there.
>
> Each AST node carries with it an index into the input token stream;  
> hidden tokens are still present in that stream, so when writing out  
> the output you can examine adjacent indicies to see if they're  
> hidden tokens that you want to output as well.  (You may need to  
> look at more than one adjacent token, since you can have consecutive  
> comments and there'll probably be whitespace tokens as well.)
>
> Not sure if this is the best approach or not, but it ought to work.
>
> Another option would be to un-hide the comments and explicitly add  
> them to the AST, but that complicates writing parsing rules quite a  
> lot, so it's probably not worth it.
>
> And I think there is also a special 'rewrite' mode you can use, but  
> I don't really know how that works, since I've never used it myself  
> (but I think the "tweak" example grammar demonstrates this).
>



More information about the antlr-interest mailing list