[antlr-interest] Unexpected CommonTokenStream.Size() result in CSharp runtime

[Chris Lambrou]

Indhu,

Thanks for that. I'll keep my tokenStream.LT(0) cludge for now. ITokenStream
doesn't have a getTokens() method, and changing my code to reference
CommonTokenStream instead of ITokenStream breaks my unit tests that use mock
objects.

FWIW, as a relative newcomer to ANTLR, it seems to me that either
CommonTokenStream isn't correctly honouring the ITokenSteam interface, or
else perhaps ITokenStream ought to be updated to formalise
CommonTokenStream's behaviour (though I'm not familiar enough with the other
ITokenStream sub-classes to be sure about this).

Cheers,

Chris

2009/4/16 Indhu Bharathi <indhu.b at s7software.com>

>  This is expected behavior only. It is designed this way so that user can
> filter for a particular channel if needed. For your task, you can use
> getTokens() method which will return a List of tokens that can be iterated.
>
> Cheers, Indhu
>
> Chris Lambrou wrote:
>
> Yesterday I was stung by some odd behaviour in CommonTokenStream, whereby I
> was trying to iterate over the token stream looking for tokens of a specific
> type. Since ITokenStream doesn't implement IEnumerable, it appears that the
> way to do this is as follows:
>
> for (int i = 0; i < tokenStream.Size(); i++)
> {
>     IToken token = tokenStream.Get(i);
>
>     //... do stuff with token...
> }
>
> However, I was finding that tokenStream.Size() returned 0, despite my token
> stream being non-empty. It seems that the underlying stream is lazily
> populated internally, and CommonTokenStream.Size() doesn't trigger a load. I
> had to invoke tokenStream.LT(0) to trigger the lazy load prior to looping
> through the tokens. Is this intended behaviour? Does is happen in all of the
> different runtimes? If so, it's very counterintuitive.
>
> Incidentally, could ITokenStream be updated to implement
> IEnumerable<IToken> please? Would others find this useful? I don't mind
> doing the work, but to whom should I submit a patch?
>
> Chris
>
> ------------------------------
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20090416/1aaed423/attachment.html