[antlr-interest] Lazy load of CommonTokenStream??

Martin Probst mail at martin-probst.com
Mon Aug 18 05:35:00 PDT 2008


>> CommonTokenStream buffers all tokens up when it is first used.
>> Unfortunately, size() does not trigger filling the buffer which seems
>> like a bug. I see no reason why it shouldn't fill the buffer, that  
>> is.
>
> I agree. In fact this might well be more intuitive, however we should
> think about any performance impact as this might be the reason it does
> not do this - in the C runtime I can take care of this by flipping
> functions after the first call, but you can't hack that in Java.

If you are concerned about the performance of these simple checks, why  
not make the TokenStream tokenize the whole input document within the  
constructor?

I know it feels a bit weird for a Java programmer to do much work in  
the constructor, but if you write a class and about every method call  
first checks if the object is correctly initialized, and the object is  
certain to be used (no benefit from lazy loading or similar), then why  
not initialize it properly in the constructor and be done with it?

That would also make the code in CommonTokenStream a lot simpler in  
many places.

Regards,
Martin


More information about the antlr-interest mailing list