[antlr-interest] FailedPredicateException leads to infinite loop - bug in the Lexer?

[Ron Hunter-Duvar]

Hi Cliff,

I reported this same problem on February 10 on this list. It's an Antlr 
bug, and my emails on it had the work around (which requires you to 
implement a custom superclass if you haven't already). If you search the 
list archives you should be able to find it.

Ron


Cliff Hudson wrote:
> I have been trying to work through an issue with an infinite loop caused
> when no tokens can be matched because a predicate has failed its test.  The
> problem appears to be in the Lexer.NextToken() (looking at CSharp2 sources)
> method, which I have copied here for reference:
>
> 		/// <summary>
> 		/// Return a token from this source; i.e., Match a token on the char stream.
> 		/// </summary>
> 		public virtual IToken NextToken()
> 		{
> 			while (true)
> 			{
> 				state.token = null;
> 				state.channel = Token.DEFAULT_CHANNEL;
> 				state.tokenStartCharIndex = input.Index;
> 				state.tokenStartCharPositionInLine = input.CharPositionInLine;
> 				state.tokenStartLine = input.Line;
> 				state.text = null;
> 				if (input.LA(1) == (int)CharStreamConstants.EOF)
> 				{
> 					return Token.EOF_TOKEN;
> 				}
> 				try
> 				{
> 					mTokens();
> 					if (state.token == null)
> 					{
> 						Emit();
> 					}
> 					else if (state.token == Token.SKIP_TOKEN)
> 					{
> 						continue;
> 					}
> 					return state.token;
> 				}
> 				catch (NoViableAltException nva) {
> 					ReportError(nva);
> 					Recover(nva); // throw out current char and try again
> 				}
> 				catch (RecognitionException re) {
> 					ReportError(re);
> 					// Match() routine has already called Recover()
> 				}
> 			}
> 		}
>
> Note the RecognitionException clause.  This is the clause which will
> catch the FailedPredicateException().  Unfortunately, because the
> FailedPredicateException gets thrown just before Match() occurs in the
> rule, Recover will *not* have been called by the rule or its callees,
> and therefore the DFA will continue to try processing the same token.
> It would appear that there should instead be a specific
> FailedPredicateException clause which does the same thing as the
> NoViableAltException clause.
>
> I have seen two other people ask questions about this error, and in
> neither case was a suitable response given.  Has this bug been fixed
> in non-released builds?  Can someone give me an up-or-down on whether
> this is a correct and appropriate fix?
>
> Thanks.
>
> - Cliff
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>   

-- 
Ron Hunter-Duvar | Software Developer V | 403-272-6580
Oracle Service Engineering
Gulf Canada Square 401 - 9th Avenue S.W., Calgary, AB, Canada T2P 3C5

All opinions expressed here are mine, and do not necessarily represent
those of my employer.