[antlr-interest] Bug in 2.7.6 with lexer option defaultErrorHandler = true

[Alexey Demakov]

From: "Terence Parr" <parrt at cs.usfca.edu>
> On Feb 15, 2006, at 5:53 AM, Alexey Demakov wrote:
> 
> > When lexer option defaultErrorHandler is set to true,
> > antlr-2.7.6 generates error handlers with call of
> > public void recover( RecognitionException ex, BitSet tokenSet )
> >
> > but this method is defined only in Parser, not in CharScanner,
> > so it causes compilation errors.
> >
> > antlr-2.7.2 "inlines" this method as
> >
> > consume();
> > consumeUntil( tokenSet );
> >
> > Workaround - define this method in lexer:
> >
> > public void recover( RecognitionException ex, BitSet tokenSet )  
> > throws CharStreamException {
> >  consume();
> >  consumeUntil( tokenSet );
> > }
> 
> I swear I fixed that but as you say...fix did not get in there. : 
> (  We're going to do a 2.7.7 soon.

I'm starting to doubt that it is the best way for recovery in lexer...
All text matched by consume() and  consumeUntil( tokenSet ) 
is appended to token.  May be it's better to set saveConsumedInput = false before consume()
and restore saveConsumedInput = true after consumeUntil()?
And how can I completely skip this malformed token?
I can't set token type to Token.SKIP in recover() and don't want 
to define custom error handlers for each lexer rule.
May be allow to return a value from recover() method?

public boolean recover( RecognitionException ex, BitSet tokenSet )  
throws CharStreamException {
  consume();
  consumeUntil( tokenSet );
  return false;
}

and generate error handler as

catch (RecognitionException ex) {
   reportError(ex);
   if( !recover(ex,_tokenSet_1) ) _type = Token.SKIP;
}

---
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com