[antlr-interest] preserving whitespace

[Donal Murtagh]

Hi Bryan,

Thanks for your response. My understanding of the article is the same as you've described. The
shortcoming I mentioned below is that "whitespace occurring in between statements [not tokens]" is
omitted.  

I think it's more accurate to characterise this behaviour as a shortcoming rather than a design
choice. If this whitespace were preserved, the client could always remove it if required, but if
it's "automatically" discarded....

I appreciate that the TokenStreamTracker object keeps *all* the tokens, but the problem is that if
you walk the tree and ask a node to print out it's children (including whitespace),
leading/tailing whitespace is omitted. 

Best Wishes,
Don

--- Bryan Ewbank <ewbank at gmail.com> wrote:

> I think you misread the article.  All whitespace is preserved by the
> TokenStreamTracker object.  The problem is that whitespace before or
> after the interesting tokens is not easily accessible.  That means
> that "a + b     " will be displayed as "a + b" - the trailing
> whitespace is not contained within the sequence of tokens tracked.
> (nor is leading whitespace).
> 
> Perhaps the TokenStreamTracker could be generous and harvest adjacent
> white-space tokens when a sequence is asked for(?), but that's really
> dependent on exactly what you want to do.
> 
> Hope this helps,
> - Bryan Ewbank
> 
> On 12/6/05, Donal Murtagh <donalmurtagh at yahoo.co.uk> wrote:
> > Hi,
> >
> > In Terence's article:
> > http://www.antlr.org/article/preserving.token.order/preserving.token.order.tml
> >
> > He describes a means of preserving the original token sequence and whitespace. A shortcoming
> of
> > his solution is that whitespace occurring in between statements is lost. Overcoming this is
> left
> > as an exercise for the reader :o)
> >
> > Have any readers figured out how to modify the solution in order to overcome this? If so, I'd
> be
> > grateful if they'd share it.
> >
> > Best Wishes,
> > Don
> >
> >
> >
> >
> >
> >
> >
> > ___________________________________________________________
> > How much free photo storage do you get? Store your holiday
> > snaps for FREE with Yahoo! Photos http://uk.photos.yahoo.com
> >
> 



	
	
		
___________________________________________________________ 
Yahoo! Messenger - NEW crystal clear PC to PC calling worldwide with voicemail http://uk.messenger.yahoo.com