[antlr-interest] token stream synchronization

Dennis Marsa drm at xilinx.com
Mon May 13 11:28:02 PDT 2002


The default exception handler generated by ANTLR includes
calls to both consume() and consumeUntil() as in:

  catch (RecognitionException& ex) {
    if( inputState->guessing == 0 ) {
      reportError(ex);
      consume();
      consumeUntil(_tokenSet_0);
    } else {
      throw;
    }
  }

The call to consume() discards the bad token that led
to the exception being thrown.  consumeUntil() discards
0 or more additional tokens until one is found that is
in the rule's follow set, or EOF is found.

Consequently, the 1st token (the one that caused the
exception to be thrown) is never compared against the
follow set.

Thus, it is never possible to resynchronize the token
stream *and* consume 0 tokens.  The first token will
*always* be discarded (by the consume() call), even if
it is a token that is in the follow set.

Is this a requirement?  Is there some reason the first
token should not be compared against the follow aet
as well?

Is there a fundamental reason that the following exception
handler (call to consume() removed) would not be as effective
as the original?  The benefit would be that synchronizations
that consume 0 tokens are now possible.

  catch (RecognitionException& ex) {
    if( inputState->guessing == 0 ) {
      reportError(ex);
      consumeUntil(_tokenSet_0);
    } else {
      throw;
    }
  }

Regards,

Dennis Marsa

 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list