[antlr-interest] Lazy load of CommonTokenStream??
Kay Röpke
kroepke at classdump.org
Sun Aug 17 12:01:19 PDT 2008
On Aug 17, 2008, at 9:50 PM, Vitaliy wrote:
> I've encountered some really interesting behavior of the
> CommonTokenStream:
>
> Right after the constructor of the parser, I'm trying to use the
> method Size() of CommonTokenStream, to retrieve the number of tokens
> in the input.
> It would return 0, no matter how many tokens actually exist in the
> input.
> BUT funny enough, if I call before that to the method LA(1), then
> everything works fine.
CommonTokenStream buffers all tokens up when it is first used.
Unfortunately, size() does not trigger filling the buffer which seems
like a bug. I see no reason why it shouldn't fill the buffer, that is.
> Could anyone please enlighten me regarding the reason?
> Can it be avoided somehow? Is there a better way than calling LA(1)?
Just call LA(1) for now, it works and doesn't hurt, after all it gives
you the next lookahead token which is a cheap operation.
Alternatively you can ask for all tokens (getTokens()) which seems
like a waste because it gives you a new copy of the list (it does it
because you can give it a tokentype filter).
If you definitely want to get rid of the LA(1) call, simply subclass
CommonTokenStream and override size() to be:
public int size() {
if ( p == -1 ) { fillBuffer(); }
return super.size();
}
cheers,
-k
--
Kay Röpke
http://classdump.org/
More information about the antlr-interest
mailing list