[antlr-interest] case sensitivity for ANTLR v3 lexers

[Olivier Dragon]

On Tue, May 16, 2006 at 09:37:02PM +0200, Martin Probst wrote:
> Hi,
> 
> >If you need a case-insensitive lexer, the end user should implement  
> >that by
> >overriding an appropriate method in the lexer and use whatever  
> >comparison is
> >appropriate for their particular needs.  There is no universal  
> >upper or
> >lower-casing function that is appropriate for every possible  
> >locale, so why
> >even go down that road?   Keep the core of Antlr simple, and just  
> >provide
> >extensibility points (with samples) where appropriate.
> 
> It depends on how common the case insensitive lexer thing is. I have  
> personally never used a case insensitive programming language since  
> PASCAL (that was case insensitive, wasn't it?). It should only be an  
> option if really a lot of people need it, otherwise allow for the  
> overriding, provide a Wiki article and a FAQ entry and that's it.  
> Maybe some more detailed Wiki article about using IBM ICU would be  
> nice, but not really needed.

I'd be forced to agree with Don here. FORTRAN 77 (which I need ANTLR
for) and SQL are but two languages I know of which are case insensitive.

I would most definetly agree with a small simple core design allowing
extensions where needed. A "default" extension could be provided with
ANTLR for an english/ascii lexer both as a convenience, as I suspect
most users will be using it, and as an example of subclassing
/extending/ifdefing for those who don't. This should obviously be
supplemented with the proper documentation as you suggested.

In any case, hard coding character handling and string messages like in
2.x in the codegen is a terrible idea. Especially if you're already
using StringTemplate: there are no excuses.

-Olivier

-- 
          __-/|    ? ?     |\-__
     __--/  /  \   (^^)   /  \  \--__
  _-/   /   /  /\ / ( )  /\  \   \   \-_
 /  /   /  /  /  (   ^^ ~  \  \  \   \  \
 / Oli Dragon    ( dragonoe at mcmaster.ca \
/  B.Eng. Sfwr   ( dragon.homelinux.org  \
/  /  /    /__--_ (   ) __--__\    \  \  \
|  /  /  _/        \_ \_       \_  \  \  |
 \/  / _/            \_ \_       \_ \  \/
  \_/ /                -\_\        \ \_/
    \/                    )         \/
                        *~
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://www.antlr.org/pipermail/antlr-interest/attachments/20060516/d2fc64b7/attachment.bin