[antlr-interest] Capturing lexer exceptions

[Jim Idle]

Michael,

I think that really, you should not have a lexer that can generate errors, in that it should be able to handle anything thrown at it and produce some sort of token or handle it in some way. Hence the ANTLR3 lexer just reports the character it sees that it doesn't know what to do with and throws it away (consumes it). 

So, if your lexer spec does not cover all the characters that might come in to it, then this is what will happen. However, you can probably achieve what you are looking for by specifying:

UNKNOWN: . {do something, or not}

as the last lexer rule, and either looking in the parser for this and reporting the error in some way yourself, or just setting the token to be skipped.

You might also find the filter mode is what you are looking for in the lexer? Check the examples for a filtering lexer.

Yet another option is to override the lexer error handler, or more specifically error display using @lexer:: ...  and just don't print anything, which should cause the character to be consumes silently. 

However, my own opinion is that you should make sure your lexer is able to process all characters and perform some specific, deliberate action such as set the token to be skipped as you then comment it as to why you are doing this and it just feels more logical to me, unless you are specifically constructing some sort of filter.

Jim





-----Original Message-----
From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Gerz, Michael
Sent: Tuesday, October 31, 2006 4:42 AM
To: antlr-interest at antlr.org
Cc: Schueller, Henriette
Subject: [antlr-interest] Capturing lexer exceptions

Terence,

I am sorry for bothering you again but my project team is still struggling with ANTLR's clever error recovery. 

Our problem is that we don't want ANTLR to do any error handling at all. This includes errors thrown by the lexer. My colleague didn't find a way to make her ANTLR parser _not_ to catch lexer exceptions.

Unfortunately, the ANTLR 3 wiki (http://www.antlr.org/wiki/display/ANTLR3/Migrating+from+ANTLR+2+to+ANTLR+3) doesn't answer the question how to ignore lexer exceptions (and you can no longer set "defaultErrorHandler=false").

Terence, could you please speak to us?

A thousand thanks in advance,

Michael

************************************************************************
Dr. Michael Gerz                                                        
FGAN e.V.                                      Phone:  +49 228 9435-414 
Department FKIE/ITF                            Fax:    +49 228 9435-685 
Neuenahrer Straße 20                           E-Mail: gerz at fgan.de     
53343 Wachtberg-Werthhoven                     WWW:    www.fgan.de     

-- 
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.19/507 - Release Date: 10/31/2006
 

-- 
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.13.19/507 - Release Date: 10/31/2006