[antlr-interest] Fwd: Re: Trying to keep whitespace in an AST

[Artem Portnoy]

I'm trying to solve the exact same problem. I found some sample code here http://www.antlr.org/article/whitespace/index.html. The problem is that the way the existing Java grammar is written, not all the tokens end up in the AST tree, such as semicolons, curly brakets, etc. The result of this is some of the comments end up getting weaved onto these "missing" tokens, and there's no way to get them back from AST. If somebody knows how to resolve this issue please share the information.

Thank you,

Artem
> Date: Fri, 8 Feb 2008 16:24:14 +0100
> From: jaluber at gmx.de
> To: antlr-interest at antlr.org
> Subject: [antlr-interest]  Fwd: Re:  Trying to keep whitespace in an AST
> 
> > So I guess I need an AST that keeps newlines and
> > comments associated 
> > with particular nodes. The pretty printing is done
> > by our code so the 
> > grammar doesn't need to do that.
> 
> Well, there are problems with the naive way of simply attaching newlines 
> and comments. The AST grammar doesn't recognize "The comments and 
> whitespace everywhere" requirement. So you can insert at every possible 
> point "(WS | COMMENT)*", but that would be ugly.
> 
> One could try to tell the parser to check for WS and COMMENT in addition 
> to the expected node, but that would interfere with syntactic 
> predicates, which don't check for theses variants. One could try to 
> change the template but I'm not sure how feasible this is. There could 
> be other problems lurking.
> 
> The way I'd go is to use channels. Put newlines and COMMENT into a 
> channel separate from the unneeded whitespace. This would reduce the 
> problem into accessing the off-channel tokens at the right places and to 
> keep them after grammar transformations. But so far I haven't checked 
> out, how one can do the latter properly.
> 
> Johannes

_________________________________________________________________
Helping your favorite cause is as easy as instant messaging. You IM, we give.
http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080208/1b1894f9/attachment.html