[antlr-interest] Lexer exception?

[Alexey Demakov]

As far as I know defaultErrorHandler = false causes lexer
to throw TokenStreamException. By default parser catches RecognitionException only,
so parser will pass TokenStreamException without any messages.

I think

line 1:1: unexpected token: 111

caused by error in parser not in lexer. It looks like default parser error message.
If 111 is valid token, there is no error in lexer. If file can't start from
this token, it is parser error.

Regards,
Alexey

-----
Alexey Demakov
TreeDL: Tree Description Language: http://treedl.sourceforge.net
RedVerst Group: http://www.unitesk.com


----- Original Message ----- 
From: "Jens Bartelheimer" <jens.bartelheimer at gmx.de>
To: "Alexey Demakov" <demakov at ispras.ru>
Cc: "antlr-interest" <antlr-interest at antlr.org>
Sent: Monday, February 20, 2006 12:48 PM
Subject: Re: [antlr-interest] Lexer exception?


> Ok, this method solves the problem if the option is set to true.
> 
> As Martin mentioned I have to set this option to false but there are no 
> additional exceptions. I think this is idependent from the bug in antlr. 
> Am I right? I also do not want to skip the malformed token, I only need 
> an exception so that I can recognize the wrong input  if  I call the 
> parser. Further I do not need the printing of the message "line 1:1: 
> unexpected token: 111". Is this only possible with the $skipToken....?
> 
> Thanks,
> Jens
> 
> Alexey Demakov wrote:
>> Hi Jens,
>>
>> Aa I wrote recently, it is the bug in antlr.
>> http://www.antlr.org/pipermail/antlr-interest/2006-February/015417.html 
>>
>>
>> Define this method in your lexer code:
>>
>> public void recover( RecognitionException ex, BitSet tokenSet ) throws 
>> CharStreamException {
>>  saveConsumedInput = false;  consume();
>>  consumeUntil( tokenSet );
>>  saveConsumedInput = true; }
>>
>> But if you want completly skip malformed token, you have to define 
>> custom exception handler
>> with $setType( Token.SKIP ) for each lexer rule. :(
>>
>> ---
>> Alexey Demakov
>> TreeDL: Tree Description Language: http://treedl.sourceforge.net
>> RedVerst Group: http://www.unitesk.com
>>
>>
>> From: "Jens Bartelheimer" <jens.bartelheimer at gmx.de>
>>> Hi Martin,
>>>
>>> thanks for your reply but if I turn off the option there are no 
>>> additional exceptions.
>>>
>>> If I turn it on explicit I have the following error in the Lexer:
>>>
>>> My error message:
>>> The method recover(RecognitionException, BitSet) is undefined for the 
>>> type UMLLexer
>>>
>>> Why? Is there a mistake in my Lexer?
>>>
>>> Jens
>>>
>>>
>>> Martin Probst wrote:
>>> >> how can I catch such an exception? I do not found something in the 
>>> >> documentation.
>>> >>
>>> >> line 1:1: unexpected token: 111
>>> >>     >
>>> > You have to turn off the default error handler, e.g. in your options
>>> > section of the lexer "defaultErrorHandler = false". Afterwards you can
>>> > catch exceptions. All of them extend ANTLRException, the manual 
>>> contains
>>> > a part called "Exception hierarchy".
>>
>>
>>