[antlr-interest] skipOffChannelTokens in CommonTokenStream
Martin Probst
mail at martin-probst.com
Wed Apr 2 07:46:57 PDT 2008
False alarm, my token was not on the right channel. But that
CommonTokenStream is really quite a confusing machine for such a
semmingly simple task...
Regards,
Martin
Am 02.04.2008 um 12:08 schrieb Martin Probst:
> Hi,
>
> I'm running into a bug where the parser will see and analyze tokens
> that are on a hidden channel, thus failing with noviablealt
> exceptions. I think the culprit is this code in
> CommonTokenStream.LT(int), around line 250.
>
> int i = p;
> int n = 1;
> // find k good tokens
> while ( n<k ) {
> // skip off-channel tokens
> i = skipOffTokenChannels(i+1); // leave p on valid token
> n++;
> }
>
> Where k is the desired lookahead, p is the current position in the
> token stream and i is the position of the actually asked token. If k
> == 1 the while loop will never execute. So if the token at k (at 1)
> is a $channel == HIDDEN token, it will still be returned, even if it
> should be ignored.
>
> I think changing the condition to "while ( n<=k )" and changing
> "skipOffTokenChannels(i+1)" to "skipOffTokenChannels(i)" should fix
> the issue.
>
> Regards,
> Martin
More information about the antlr-interest
mailing list