[antlr-interest] Lazy load of CommonTokenStream??

Vitaliy Vitaliy at dbsophic.com
Mon Aug 18 01:49:21 PDT 2008


Thanks for the quick responses!
I'm glad I was able to contribute somehow to ANTLR :)

BTW, could LA(1) fail somehow - if there is no next token,
or if it goes beyond the last token, or something like that?

Thanks,
Vitaliy

-----Original Message-----
From: Kay Röpke [mailto:kroepke at classdump.org]
Sent: Sunday, August 17, 2008 21:01
To: Vitaliy
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Lazy load of CommonTokenStream??


On Aug 17, 2008, at 9:50 PM, Vitaliy wrote:

> I've encountered some really interesting behavior of the
> CommonTokenStream:
>
> Right after the constructor of the parser, I'm trying to use the
> method Size() of CommonTokenStream, to retrieve the number of tokens
> in the input.
> It would return 0, no matter how many tokens actually exist in the
> input.
> BUT funny enough, if I call before that to the method LA(1), then
> everything works fine.

CommonTokenStream buffers all tokens up when it is first used.
Unfortunately, size() does not trigger filling the buffer which seems
like a bug. I see no reason why it shouldn't fill the buffer, that is.

> Could anyone please enlighten me regarding the reason?
> Can it be avoided somehow?  Is there a better way than calling LA(1)?


Just call LA(1) for now, it works and doesn't hurt, after all it gives
you the next lookahead token which is a cheap operation.
Alternatively you can ask for all tokens (getTokens()) which seems
like a waste because it gives you a new copy of the list (it does it
because you can give it a tokentype filter).

If you definitely want to get rid of the LA(1) call, simply subclass
CommonTokenStream and override size() to be:

public int size() {
        if ( p == -1 ) { fillBuffer(); }
        return super.size();
}

cheers,
-k

--
Kay Röpke
http://classdump.org/











__________ Information from ESET NOD32 Antivirus, version of virus signature database 3362 (20080817) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com



__________ Information from ESET NOD32 Antivirus, version of virus signature database 3363 (20080818) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com



More information about the antlr-interest mailing list