[antlr-interest] token stream synchronization

[Terence Parr]

On Monday, May 13, 2002, at 11:28  AM, Dennis Marsa wrote:

> The default exception handler generated by ANTLR includes
> calls to both consume() and consumeUntil() as in:
>
>   catch (RecognitionException& ex) {
>     if( inputState->guessing == 0 ) {
>       reportError(ex);
>       consume();
>       consumeUntil(_tokenSet_0);
>     } else {
>       throw;
>     }
>   }
>
> The call to consume() discards the bad token that led
> to the exception being thrown.  consumeUntil() discards
> 0 or more additional tokens until one is found that is
> in the rule's follow set, or EOF is found.
>
> Consequently, the 1st token (the one that caused the
> exception to be thrown) is never compared against the
> follow set.
>
> Thus, it is never possible to resynchronize the token
> stream *and* consume 0 tokens.  The first token will
> *always* be discarded (by the consume() call), even if
> it is a token that is in the follow set.
>
> Is this a requirement?  Is there some reason the first
> token should not be compared against the follow aet
> as well?

Yep, you can easily get into an infinite loop.  ANTLR is guaranteed to 
terminate if I have a consume at each error.  I suppose I could have 
another means that tracked whether I'd been at a space w/o consuming 
but that is harder.  Are you getting bad error sync?

BTW, I saw the infinite loop problem when building the C++ parser with 
the folks at NeXT, hence, I added the "must terminate" functionality :)

Ter
--
Co-founder, http://www.jguru.com
Creator, ANTLR Parser Generator: http://www.antlr.org


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/