[antlr-interest] Unexpected CommonTokenStream.Size() result in CSharp runtime

[Chris Lambrou]

Yesterday I was stung by some odd behaviour in CommonTokenStream, whereby I
was trying to iterate over the token stream looking for tokens of a specific
type. Since ITokenStream doesn't implement IEnumerable, it appears that the
way to do this is as follows:

for (int i = 0; i < tokenStream.Size(); i++)
{
    IToken token = tokenStream.Get(i);

    //... do stuff with token...
}

However, I was finding that tokenStream.Size() returned 0, despite my token
stream being non-empty. It seems that the underlying stream is lazily
populated internally, and CommonTokenStream.Size() doesn't trigger a load. I
had to invoke tokenStream.LT(0) to trigger the lazy load prior to looping
through the tokens. Is this intended behaviour? Does is happen in all of the
different runtimes? If so, it's very counterintuitive.

Incidentally, could ITokenStream be updated to implement IEnumerable<IToken>
please? Would others find this useful? I don't mind doing the work, but to
whom should I submit a patch?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090416/0b8ef9f9/attachment.html