[antlr-interest] Fwd: Re: Trying to keep whitespace in an AST

[Johannes Luber]

> So I guess I need an AST that keeps newlines and
> comments associated 
> with particular nodes. The pretty printing is done
> by our code so the 
> grammar doesn't need to do that.

Well, there are problems with the naive way of simply attaching newlines 
and comments. The AST grammar doesn't recognize "The comments and 
whitespace everywhere" requirement. So you can insert at every possible 
point "(WS | COMMENT)*", but that would be ugly.

One could try to tell the parser to check for WS and COMMENT in addition 
to the expected node, but that would interfere with syntactic 
predicates, which don't check for theses variants. One could try to 
change the template but I'm not sure how feasible this is. There could 
be other problems lurking.

The way I'd go is to use channels. Put newlines and COMMENT into a 
channel separate from the unneeded whitespace. This would reduce the 
problem into accessing the off-channel tokens at the right places and to 
keep them after grammar transformations. But so far I haven't checked 
out, how one can do the latter properly.

Johannes