[antlr-interest] Possible bug in C-target runtime related to off-channel tokens

[troy runkel]

The grammar I'm working on is configured to write comments and
whitespace to the hidden token channel.  I found that if the text I'm
trying to parse begins with one of these hidden tokens the token is
sent to the parser instead of being skipped.

It looks like the problem is in the fillBuffer function in
antlr3tokenstream.cpp.  At the end of fillBuffer we call
skipOffTokenChannels and then initialize the
tokenStream->tstream->istream->cachedSize variable.  However,
skipOffTokenChannels refers to
tokenStream->tstream->istream->cachedSize.  In my test case the cached
size was 0 so skipOffTokenChannels failed to skip the hidden channel
tokens.

I fixed the problem by simply moving the call to skipOffTokenChannels
after setting the tokenStream->tstream->istream->cachedSize variable.

I believe you'll only see this problem if you also set
discardOffChannelToks to FALSE in the token stream (which is what I'm
doing).

Troy Runkel