[antlr-interest] skipOffChannelTokens in CommonTokenStream

Martin Probst mail at martin-probst.com
Wed Apr 2 07:46:57 PDT 2008


False alarm, my token was not on the right channel. But that  
CommonTokenStream is really quite a confusing machine for such a  
semmingly simple task...

Regards,
Martin

Am 02.04.2008 um 12:08 schrieb Martin Probst:
> Hi,
>
> I'm running into a bug where the parser will see and analyze tokens  
> that are on a hidden channel, thus failing with noviablealt  
> exceptions. I think the culprit is this code in  
> CommonTokenStream.LT(int), around line 250.
>
> 		int i = p;
> 		int n = 1;
> 		// find k good tokens
> 		while ( n<k ) {
> 			// skip off-channel tokens
> 			i = skipOffTokenChannels(i+1); // leave p on valid token
> 			n++;
> 		}
>
> Where k is the desired lookahead, p is the current position in the  
> token stream and i is the position of the actually asked token. If k  
> == 1 the while loop will never execute. So if the token at k (at 1)  
> is a $channel == HIDDEN token, it will still be returned, even if it  
> should be ignored.
>
> I think changing the condition to "while ( n<=k )" and changing  
> "skipOffTokenChannels(i+1)" to "skipOffTokenChannels(i)" should fix  
> the issue.
>
> Regards,
> Martin



More information about the antlr-interest mailing list