[antlr-interest] Unexpected CommonTokenStream.Size() result in CSharp runtime

Indhu Bharathi indhu.b at s7software.com
Thu Apr 16 02:16:36 PDT 2009


This is expected behavior only. It is designed this way so that user can 
filter for a particular channel if needed. For your task, you can use 
getTokens() method which will return a List of tokens that can be iterated.

Cheers, Indhu

Chris Lambrou wrote:
> Yesterday I was stung by some odd behaviour in CommonTokenStream, 
> whereby I was trying to iterate over the token stream looking for 
> tokens of a specific type. Since ITokenStream doesn't implement 
> IEnumerable, it appears that the way to do this is as follows:
>
> for (int i = 0; i < tokenStream.Size(); i++)
> {
>     IToken token = tokenStream.Get(i);
>
>     //... do stuff with token...
> }
>
> However, I was finding that tokenStream.Size() returned 0, despite my 
> token stream being non-empty. It seems that the underlying stream is 
> lazily populated internally, and CommonTokenStream.Size() doesn't 
> trigger a load. I had to invoke tokenStream.LT(0) to trigger the lazy 
> load prior to looping through the tokens. Is this intended behaviour? 
> Does is happen in all of the different runtimes? If so, it's very 
> counterintuitive.
>
> Incidentally, could ITokenStream be updated to implement 
> IEnumerable<IToken> please? Would others find this useful? I don't 
> mind doing the work, but to whom should I submit a patch?
>
> Chris
> ------------------------------------------------------------------------
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090416/517c9c1f/attachment.html 


More information about the antlr-interest mailing list