[antlr-interest] FailedPredicateException leads to infinite loop - bug in the Lexer?
Cliff Hudson
cliff.s.hudson at gmail.com
Mon Mar 29 20:36:05 PDT 2010
And incidentally I have tried to make this change using a snapshot of
sources obtained from Fisheye, but it appears the code has changed in such a
way that the lexer and parser produced by compiling with the Antlr 3.2
toolset is no longer compatible (for instance, IIntToken.Index is now a
property, not a method.)
On Mon, Mar 29, 2010 at 8:19 PM, Cliff Hudson <cliff.s.hudson at gmail.com>wrote:
> I have been trying to work through an issue with an infinite loop caused
> when no tokens can be matched because a predicate has failed its test. The
> problem appears to be in the Lexer.NextToken() (looking at CSharp2 sources)
> method, which I have copied here for reference:
>
> /// <summary>
> /// Return a token from this source; i.e., Match a token on the char stream.
> /// </summary>
> public virtual IToken NextToken()
> {
> while (true)
> {
> state.token = null;
> state.channel = Token.DEFAULT_CHANNEL;
> state.tokenStartCharIndex = input.Index;
> state.tokenStartCharPositionInLine = input.CharPositionInLine;
> state.tokenStartLine = input.Line;
> state.text = null;
> if (input.LA(1) == (int)CharStreamConstants.EOF)
> {
> return Token.EOF_TOKEN;
> }
> try
> {
> mTokens();
> if (state.token == null)
> {
> Emit();
> }
> else if (state.token == Token.SKIP_TOKEN)
> {
> continue;
> }
> return state.token;
> }
> catch (NoViableAltException nva) {
> ReportError(nva);
> Recover(nva); // throw out current char and try again
> }
> catch (RecognitionException re) {
> ReportError(re);
> // Match() routine has already called Recover()
> }
> }
> }
>
> Note the RecognitionException clause. This is the clause which will catch the FailedPredicateException(). Unfortunately, because the FailedPredicateException gets thrown just before Match() occurs in the rule, Recover will *not* have been called by the rule or its callees, and therefore the DFA will continue to try processing the same token. It would appear that there should instead be a specific FailedPredicateException clause which does the same thing as the NoViableAltException clause.
>
> I have seen two other people ask questions about this error, and in neither case was a suitable response given. Has this bug been fixed in non-released builds? Can someone give me an up-or-down on whether this is a correct and appropriate fix?
>
> Thanks.
>
> - Cliff
>
>
More information about the antlr-interest
mailing list