[antlr-interest] skipOffChannelTokens in CommonTokenStream

Martin Probst mail at martin-probst.com
Wed Apr 2 03:08:28 PDT 2008


Hi,

I'm running into a bug where the parser will see and analyze tokens  
that are on a hidden channel, thus failing with noviablealt  
exceptions. I think the culprit is this code in  
CommonTokenStream.LT(int), around line 250.

		int i = p;
		int n = 1;
		// find k good tokens
		while ( n<k ) {
			// skip off-channel tokens
			i = skipOffTokenChannels(i+1); // leave p on valid token
			n++;
		}

Where k is the desired lookahead, p is the current position in the  
token stream and i is the position of the actually asked token. If k  
== 1 the while loop will never execute. So if the token at k (at 1) is  
a $channel == HIDDEN token, it will still be returned, even if it  
should be ignored.

I think changing the condition to "while ( n<=k )" and changing  
"skipOffTokenChannels(i+1)" to "skipOffTokenChannels(i)" should fix  
the issue.

Regards,
Martin


More information about the antlr-interest mailing list