[antlr-interest] Fwd: Re: Trying to keep whitespace in an AST

[Artem Portnoy]

Johannes,

I have no idea how to do the same thing in Antlr v3. I couldn't find much information or examples. This is the only thing I found and I couldn't make much sense of it http://www.antlr.org/wiki/pages/viewpage.action?pageId=1057. It just sounds too vague for me with what little understanding of antlr I've got. In my case, I do not care about whitespace. All I need is the ability to access the comments.

The way TokenStreamHiddenTokenFilter works in Antlr 2 is that it looks for hidden tokens, such as comments and whitespace and sets them onto the preceding or trailing "regular" token. Because it has no knowledge of which tokens are bound for inclusion in the AST tree I'm faced with the problem I described before.

Regards,
Artem
> Date: Fri, 8 Feb 2008 17:54:47 +0100
> From: jaluber at gmx.de
> To: tema75 at hotmail.com
> CC: antlr-interest at antlr.org
> Subject: Re: [antlr-interest] Fwd: Re:  Trying to keep whitespace in an AST
> 
> Artem Portnoy schrieb:
> > I'm trying to solve the exact same problem. I found some sample code 
> > here http://www.antlr.org/article/whitespace/index.html. The problem is 
> > that the way the existing Java grammar is written, not all the tokens 
> > end up in the AST tree, such as semicolons, curly brakets, etc. The 
> > result of this is some of the comments end up getting weaved onto these 
> > "missing" tokens, and there's no way to get them back from AST. If 
> > somebody knows how to resolve this issue please share the information.
> 
> The description is still for ANTLR 2.7.7. Did you manage to translate 
> into ANTLR 3 syntax? I am nonetheless wondering, why you get the pure 
> syntactical tokens with the whitespace. My understanding is that each 
> token includes only the text it actually matched. Did you check the 
> "intermediate" tokens for being only whitespace and comments? If that 
> doesn't work the only option is to include the neglected tokens into the 
> AST.
> 
> Johannes
> 

_________________________________________________________________
Helping your favorite cause is as easy as instant messaging. You IM, we give.
http://im.live.com/Messenger/IM/Home/?source=text_hotmail_join
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20080208/fa1ba8ba/attachment.html