[antlr-interest] FailedPredicateException leads to infinite loop - bug in the Lexer?

Cliff Hudson cliff.s.hudson at gmail.com
Tue Mar 30 10:31:46 PDT 2010


I've been all over the archives, but perhaps my search terms were
inadequate.  I'll look again with that date in mind.  Thanks.

On Tue, Mar 30, 2010 at 8:11 AM, Ron Hunter-Duvar <
ron.hunter-duvar at oracle.com> wrote:

> Hi Cliff,
>
> I reported this same problem on February 10 on this list. It's an Antlr
> bug, and my emails on it had the work around (which requires you to
> implement a custom superclass if you haven't already). If you search the
> list archives you should be able to find it.
>
> Ron
>
>
> Cliff Hudson wrote:
>
>> I have been trying to work through an issue with an infinite loop caused
>> when no tokens can be matched because a predicate has failed its test.
>>  The
>> problem appears to be in the Lexer.NextToken() (looking at CSharp2
>> sources)
>> method, which I have copied here for reference:
>>
>>                /// <summary>
>>                /// Return a token from this source; i.e., Match a token on
>> the char stream.
>>                /// </summary>
>>                public virtual IToken NextToken()
>>                {
>>                        while (true)
>>                        {
>>                                state.token = null;
>>                                state.channel = Token.DEFAULT_CHANNEL;
>>                                state.tokenStartCharIndex = input.Index;
>>                                state.tokenStartCharPositionInLine =
>> input.CharPositionInLine;
>>                                state.tokenStartLine = input.Line;
>>                                state.text = null;
>>                                if (input.LA(1) ==
>> (int)CharStreamConstants.EOF)
>>                                {
>>                                        return Token.EOF_TOKEN;
>>                                }
>>                                try
>>                                {
>>                                        mTokens();
>>                                        if (state.token == null)
>>                                        {
>>                                                Emit();
>>                                        }
>>                                        else if (state.token ==
>> Token.SKIP_TOKEN)
>>                                        {
>>                                                continue;
>>                                        }
>>                                        return state.token;
>>                                }
>>                                catch (NoViableAltException nva) {
>>                                        ReportError(nva);
>>                                        Recover(nva); // throw out current
>> char and try again
>>                                }
>>                                catch (RecognitionException re) {
>>                                        ReportError(re);
>>                                        // Match() routine has already
>> called Recover()
>>                                }
>>                        }
>>                }
>>
>> Note the RecognitionException clause.  This is the clause which will
>> catch the FailedPredicateException().  Unfortunately, because the
>> FailedPredicateException gets thrown just before Match() occurs in the
>> rule, Recover will *not* have been called by the rule or its callees,
>> and therefore the DFA will continue to try processing the same token.
>> It would appear that there should instead be a specific
>> FailedPredicateException clause which does the same thing as the
>> NoViableAltException clause.
>>
>> I have seen two other people ask questions about this error, and in
>> neither case was a suitable response given.  Has this bug been fixed
>> in non-released builds?  Can someone give me an up-or-down on whether
>> this is a correct and appropriate fix?
>>
>> Thanks.
>>
>> - Cliff
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>>
>>
>
> --
> Ron Hunter-Duvar | Software Developer V | 403-272-6580
> Oracle Service Engineering
> Gulf Canada Square 401 - 9th Avenue S.W., Calgary, AB, Canada T2P 3C5
>
> All opinions expressed here are mine, and do not necessarily represent
> those of my employer.
>
>


More information about the antlr-interest mailing list