[stringtemplate-interest] Re: [antlr-interest] StringTemplate + preserving comments

Sohail Somani sohail at taggedtype.net
Tue Nov 8 22:24:49 PST 2005


On Tue, 2005-08-11 at 22:19 -0800, Terence Parr wrote:
> On Nov 8, 2005, at 10:10 PM, Sohail Somani wrote:
> 
> > On Tue, 2005-08-11 at 21:54 -0800, Terence Parr wrote:
> >> On Nov 8, 2005, at 9:51 PM, Sohail Somani wrote:
> >>>
> >>> This is quite possible. The language I'm parsing is very whitespace
> >>> happy (sometimes!). No this was not fun to parse. So it could be  
> >>> that
> >>> the lexer passes:
> >>>
> >>> COMMENT EOL SOME_THING
> >>>
> >>> And of course when I try to print the text of SOME_THING, there  
> >>> is no
> >>> hidden before.
> >>>
> >>> Any suggestions?
> >>
> >> Yes.  Make that IF a WHILE loop to print ALL hidden tokens between
> >> your token and the previous :)  It chains them all together :)
> >
> > Ah, but isn't this information available only in the tokenstream  
> > and not
> > the token? Or am I missing something...
> >
> > If I try to do SOME_THING.getText(), getHiddenBefore() will be null
> > because there is no hidden token before (as it is EOL)... So I don't
> > know where to get this chain from.
> 
> Each real token (in v2) will have a linked list between it and the  
> previous real token with all hidden tokens.  If you send newlines and  
> comments to the parser as hidden, there will be a linked list (right  
> to left) from your token to the previous token.  Are you stripping  
> out newlines perhaps?

Nope, newlines are being sent to the parser as well (although EOL+
becomes EOL). Where are the fields for this linked list in the Token? I
can't see it in the code API on the web nor in the source (*Token.java).

PS: Thanks for your help!



More information about the antlr-interest mailing list