[antlr-interest] ANTLR 3 & hidden token management?

[Terence Parr]

On Aug 6, 2006, at 10:10 AM, David Holroyd wrote:

> ANTLR 2 provides CommonHiddenStreamToken et al, which I try to use for
> layout-preserving code rewriting.  I've just been looking at v3 to see
> if manipulating ASTs containing hidden tokens has become any  
> easier, but
> I don't see equivalent classes.
>
> Is this because there's a radically different way to do this which  
> I've
> not spotted, or because it's just not implemented yet?

Hi Dave,

ANTLR v3 in a sense sends all tokens to the parser, with the hidden  
tokens on different "channels".  this is done with a simple channel  
number in each token object.  The commonTokenStream objct filters for  
only those tokens of interest to the parser, but keeps all tokens  
around.  At any point, you can walk forwards/backwards in the buffer  
looking for stuff of interest on other channels.

As for trees, I automatically track start/stop token indexes for all  
rule invocations and hence I know the precise range of tokens  
associated with each rule tree construction.  This includes all  
hidden channels simply because they are in range.  Now, it does *not*  
track whitespace etc... before/after the rule...those will presumably  
be encompassed by other rule's AST construction.

Just added to faq:

http://www.antlr.org/wiki/pages/viewpage.action?pageId=1057

Ter