[antlr-interest] Duplicating and modifying portions of the input

[Gavin Lambert]

At 04:11 11/06/2008, Geraint North wrote:
 >It seems that if the simple deep copy of the AST that I am
 >performing would also preserve the hidden tokens (does it?),
 >then I should just be able to feed to resulting expanded tree
 >into a "null" rewriter, and that should produce the expanded
 >test, comments and all - is that correct, or do I have to do
 >something to ensure that the token stream is also deep copied?

Hidden tokens will usually not make it into the AST in the first 
place, so copying the tree won't help you there.

Each AST node carries with it an index into the input token 
stream; hidden tokens are still present in that stream, so when 
writing out the output you can examine adjacent indicies to see if 
they're hidden tokens that you want to output as well.  (You may 
need to look at more than one adjacent token, since you can have 
consecutive comments and there'll probably be whitespace tokens as 
well.)

Not sure if this is the best approach or not, but it ought to 
work.

Another option would be to un-hide the comments and explicitly add 
them to the AST, but that complicates writing parsing rules quite 
a lot, so it's probably not worth it.

And I think there is also a special 'rewrite' mode you can use, 
but I don't really know how that works, since I've never used it 
myself (but I think the "tweak" example grammar demonstrates this).