[antlr-interest] Capturing lexer exceptions

Gerz, Michael Gerz at fgan.de
Wed Nov 1 22:11:29 PST 2006


Dear Jim,

thank you very much!

What you propose (adding an "ELSE" rule to the lexer) sounds like a great idea. I will have a look at this solution.

Best regards,
Michael

************************************************************************
Dr. Michael Gerz                                                        
FGAN e.V.                                      Phone:  +49 228 9435-414 
Department FKIE/ITF                            Fax:    +49 228 9435-685 
Neuenahrer Straße 20                           E-Mail: gerz at fgan.de     
53343 Wachtberg-Werthhoven                     WWW:    www.fgan.de      

> -----Ursprüngliche Nachricht-----
> Von: Jim Idle [mailto:jimi at intersystems.com] 
> Gesendet: Dienstag, 31. Oktober 2006 21:44
> An: Gerz, Michael; antlr-interest at antlr.org
> Cc: Schueller, Henriette
> Betreff: RE: [antlr-interest] Capturing lexer exceptions
> 
> Michael,
> 
> I think that really, you should not have a lexer that can 
> generate errors, in that it should be able to handle anything 
> thrown at it and produce some sort of token or handle it in 
> some way. Hence the ANTLR3 lexer just reports the character 
> it sees that it doesn't know what to do with and throws it 
> away (consumes it). 
> 
> So, if your lexer spec does not cover all the characters that 
> might come in to it, then this is what will happen. However, 
> you can probably achieve what you are looking for by specifying:
> 
> UNKNOWN: . {do something, or not}
> 
> as the last lexer rule, and either looking in the parser for 
> this and reporting the error in some way yourself, or just 
> setting the token to be skipped.
> 
> You might also find the filter mode is what you are looking 
> for in the lexer? Check the examples for a filtering lexer.
> 
> Yet another option is to override the lexer error handler, or 
> more specifically error display using @lexer:: ...  and just 
> don't print anything, which should cause the character to be 
> consumes silently. 
> 
> However, my own opinion is that you should make sure your 
> lexer is able to process all characters and perform some 
> specific, deliberate action such as set the token to be 
> skipped as you then comment it as to why you are doing this 
> and it just feels more logical to me, unless you are 
> specifically constructing some sort of filter.
> 
> Jim


More information about the antlr-interest mailing list