[antlr-interest] Lazy load of CommonTokenStream??

[Kay Röpke]

On Aug 17, 2008, at 9:50 PM, Vitaliy wrote:

> I've encountered some really interesting behavior of the  
> CommonTokenStream:
>
> Right after the constructor of the parser, I'm trying to use the  
> method Size() of CommonTokenStream, to retrieve the number of tokens  
> in the input.
> It would return 0, no matter how many tokens actually exist in the  
> input.
> BUT funny enough, if I call before that to the method LA(1), then   
> everything works fine.

CommonTokenStream buffers all tokens up when it is first used.  
Unfortunately, size() does not trigger filling the buffer which seems  
like a bug. I see no reason why it shouldn't fill the buffer, that is.

> Could anyone please enlighten me regarding the reason?
> Can it be avoided somehow?  Is there a better way than calling LA(1)?


Just call LA(1) for now, it works and doesn't hurt, after all it gives  
you the next lookahead token which is a cheap operation.
Alternatively you can ask for all tokens (getTokens()) which seems  
like a waste because it gives you a new copy of the list (it does it  
because you can give it a tokentype filter).

If you definitely want to get rid of the LA(1) call, simply subclass  
CommonTokenStream and override size() to be:

public int size() {
	if ( p == -1 ) { fillBuffer(); }
	return super.size();
}

cheers,
-k

-- 
Kay Röpke
http://classdump.org/