[antlr-interest] Re: Problems with Unicode support in ANTLR

Thu May 16 20:10:46 PDT 2002

> Perhaps I am missing the thrust of this exchange, but
> my requirement for Unicode in Antlr is to support an English 
command set and
> foreign language (initially Japanese)
> variable names.
> This talk of character blocks seems too restrictive.
> I need full Unicode support to cover the possible variable names.

ANTLR allows you to set the charVocabulary to include almost all 
Unicode character values (the exceptions being \uFFFF - used for EOF -
 and perhaps \u0000..\u0002).

The discussion is about making ANTLR smarter so that it understands 
standard character groupings so that ANTLR .g files may be more 
compact (and understandable). There might also be a performance 
benefit (or deficit).

This is very important IMO. I have included part of the definition 
required for a sample Unicode category if done manually in an ANTLR 
definition file. It is auto-generated by a tool I wrote and may 
contain errors ;-)

I believe it illustrates the value of these proposals quite nicely.

Micheal

==========================================================

Unicode Category or Class: Lu

| '\u0041'..'\u005A' 
| '\u00C0'..'\u00D6' 
| '\u00D8'..'\u00DE' 
| '\u0100' 
| '\u0102' 
| '\u0104' 
| '\u0106' 
| '\u0108' 
| '\u010A' 
| '\u010C' 
| '\u010E' 
| '\u0110' 
| '\u0112' 
| '\u0114' 
| '\u0116' 
| '\u0118' 
| '\u011A' 
| '\u011C' 
| '\u011E' 
| '\u0120' 
| '\u0122' 
| '\u0124' 
| '\u0126' 
| '\u0128' 
| '\u012A' 
| '\u012C' 
| '\u012E' 
| '\u0130' 
| '\u0132' 
| '\u0134' 
| '\u0136' 
| '\u0139' 
| '\u013B' 
| '\u013D' 
| '\u013F' 
| '\u0141' 
| '\u0143' 
| '\u0145' 
| '\u0147' 
| '\u014A' 
| '\u014C' 
| '\u014E' 
| '\u0150' 
| '\u0152' 
| '\u0154' 
| '\u0156' 
| '\u0158' 
| '\u015A' 
| '\u015C' 
| '\u015E' 
| '\u0160' 
| '\u0162' 
| '\u0164' 
| '\u0166' 
| '\u0168' 
| '\u016A' 
| '\u016C' 
| '\u016E' 
| '\u0170' 
| '\u0172' 
| '\u0174' 
| '\u0176' 
| '\u0178'..'\u0179' 
| '\u017B' 
| '\u017D' 
| '\u0181'..'\u0182' 
| '\u0184' 
| '\u0186'..'\u0187' 
| '\u0189'..'\u018B' 
| '\u018E'..'\u0191' 
| '\u0193'..'\u0194' 
| '\u0196'..'\u0198' 
| '\u019C'..'\u019D' 
| '\u019F'..'\u01A0' 
| '\u01A2' 
| '\u01A4' 
| '\u01A6'..'\u01A7' 
| '\u01A9' 
| '\u01AC' 
| '\u01AE'..'\u01AF' 
| '\u01B1'..'\u01B3' 
| '\u01B5' 
| '\u01B7'..'\u01B8' 
| '\u01BC' 
| '\u01C4' 
| '\u01C7' 
| '\u01CA' 
| '\u01CD' 
| '\u01CF' 
| '\u01D1' 
| '\u01D3' 
| '\u01D5' 
| '\u01D7' 
| '\u01D9' 
| '\u01DB' 
| '\u01DE' 
| '\u01E0' 
| '\u01E2' 
| '\u01E4' 
| '\u01E6' 
| '\u01E8' 
| '\u01EA' 
| '\u01EC' 
| '\u01EE' 
| '\u01F1' 
| '\u01F4' 
| '\u01F6'..'\u01F8' 
| '\u01FA' 
| '\u01FC' 
| '\u01FE' 
| '\u0200' 
| '\u0202' 
| '\u0204' 
| '\u0206' 
| '\u0208' 
| '\u020A' 
| '\u020C' 
| '\u020E' 
| '\u0210' 
| '\u0212' 
| '\u0214' 
| '\u0216' 
| '\u0218' 
| '\u021A' 
| '\u021C' 
| '\u021E' 
| '\u0220' 
| '\u0222' 
| '\u0224' 
| '\u0226' 
| '\u0228' 
| '\u022A' 
| '\u022C' 
| '\u022E' 
| '\u0230' 
| '\u0232' 
| '\u0386' 
| '\u0388'..'\u038A' 
| '\u038C' 
| '\u038E'..'\u038F' 
| '\u0391'..'\u03A1' 
| '\u03A3'..'\u03AB' 
| '\u03D2'..'\u03D4' 
| '\u03D8' 
| '\u03DA' 
| '\u03DC' 
| '\u03DE' 
| '\u03E0' 
| '\u03E2' 
| '\u03E4' 
| '\u03E6' 
| '\u03E8' 
| '\u03EA' 
| '\u03EC' 
| '\u03EE' 
| '\u03F4' 
| '\u0400'..'\u042F' 
| '\u0460' 
| '\u0462' 
| '\u0464' 
| '\u0466' 
| '\u0468' 
| '\u046A' 
| '\u046C' 
| '\u046E' 
| '\u0470' 
| '\u0472' 
| '\u0474' 
| '\u0476' 
| '\u0478' 
| '\u047A' 
| '\u047C' 
| '\u047E' 
| '\u0480' 
| '\u048A' 
| '\u048C' 
| '\u048E' 
| '\u0490' 
| '\u0492' 
| '\u0494' 
| '\u0496' 
| '\u0498' 
| '\u049A' 
| '\u049C' 
| '\u049E' 
| '\u04A0' 
| '\u04A2' 
| '\u04A4' 
| '\u04A6' 
| '\u04A8' 
| '\u04AA' 
| '\u04AC' 
| '\u04AE' 
| '\u04B0' 
| '\u04B2' 
| '\u04B4' 
| '\u04B6' 
| '\u04B8' 
| '\u04BA' 
| '\u04BC' 
| '\u04BE' 
| '\u04C0'..'\u04C1' 
| '\u04C3' 
| '\u04C5' 
| '\u04C7' 
| '\u04C9' 
| '\u04CB' 
| '\u04CD' 
| '\u04D0' 
| '\u04D2' 
| '\u04D4' 
| '\u04D6' 
| '\u04D8' 
| '\u04DA' 
| '\u04DC' 
| '\u04DE' 
| '\u04E0' 
| '\u04E2' 
| '\u04E4' 
| '\u04E6' 
| '\u04E8' 
| '\u04EA' 
| '\u04EC' 
| '\u04EE' 
| '\u04F0' 
| '\u04F2' 
| '\u04F4' 
| '\u04F8' 
| '\u0500' 
| '\u0502' 
| '\u0504' 
| '\u0506' 
| '\u0508' 
| '\u050A' 
| '\u050C' 
| '\u050E' 
| '\u0531'..'\u0556' 
| '\u10A0'..'\u10C5' 
| '\u1E00' 
| '\u1E02' 
| '\u1E04' 
| '\u1E06' 
| '\u1E08' 
| '\u1E0A' 
| '\u1E0C' 
| '\u1E0E' 
| '\u1E10' 
| '\u1E12' 
| '\u1E14' 
| '\u1E16' 
| '\u1E18' 
| '\u1E1A' 
| '\u1E1C' 
| '\u1E1E' 
| '\u1E20' 
| '\u1E22' 
| '\u1E24' 
| '\u1E26' 
| '\u1E28' 
| '\u1E2A' 
| '\u1E2C' 
| '\u1E2E' 
| '\u1E30' 
| '\u1E32' 
| '\u1E34' 
| '\u1E36' 
| '\u1E38' 
| '\u1E3A' 
| '\u1E3C' 
| '\u1E3E' 
| '\u1E40' 
| '\u1E42' 
| '\u1E44' 
| '\u1E46' 
| '\u1E48' 
| '\u1E4A' 
| '\u1E4C' 
| '\u1E4E' 
| '\u1E50' 
| '\u1E52' 
| '\u1E54' 
| '\u1E56' 
| '\u1E58' 
| '\u1E5A' 
| '\u1E5C' 
| '\u1E5E' 
| '\u1E60' 
| '\u1E62' 
| '\u1E64' 
| '\u1E66' 
| '\u1E68' 
| '\u1E6A' 
| '\u1E6C' 
| '\u1E6E' 
| '\u1E70' 
| '\u1E72' 
| '\u1E74' 
| '\u1E76' 
| '\u1E78' 
| '\u1E7A' 
| '\u1E7C' 
| '\u1E7E' 
| '\u1E80' 
| '\u1E82' 
| '\u1E84' 
| '\u1E86' 
| '\u1E88' 
| '\u1E8A' 
| '\u1E8C' 
| '\u1E8E' 
| '\u1E90' 
| '\u1E92' 
| '\u1E94' 
| '\u1EA0' 
| '\u1EA2' 
| '\u1EA4' 
| '\u1EA6' 
| '\u1EA8' 
| '\u1EAA' 
| '\u1EAC' 
| '\u1EAE' 
| '\u1EB0' 
| '\u1EB2' 
| '\u1EB4' 
| '\u1EB6' 
| '\u1EB8' 
| '\u1EBA' 
| '\u1EBC' 
| '\u1EBE' 
| '\u1EC0' 
| '\u1EC2' 
| '\u1EC4' 
| '\u1EC6' 
| '\u1EC8' 
| '\u1ECA' 
| '\u1ECC' 
| '\u1ECE' 
| '\u1ED0' 
| '\u1ED2' 
| '\u1ED4' 
| '\u1ED6' 
| '\u1ED8' 
| '\u1EDA' 
| '\u1EDC' 
| '\u1EDE' 
| '\u1EE0' 
| '\u1EE2' 
| '\u1EE4' 
| '\u1EE6' 
| '\u1EE8' 
| '\u1EEA' 
| '\u1EEC' 
| '\u1EEE' 
| '\u1EF0' 
| '\u1EF2' 
| '\u1EF4' 
| '\u1EF6' 
| '\u1EF8' 
| '\u1F08'..'\u1F0F' 
| '\u1F18'..'\u1F1D' 
| '\u1F28'..'\u1F2F' 
| '\u1F38'..'\u1F3F' 
| '\u1F48'..'\u1F4D' 
| '\u1F59' 
| '\u1F5B' 
| '\u1F5D' 
| '\u1F5F' 
| '\u1F68'..'\u1F6F' 
| '\u1FB8'..'\u1FBB' 
| '\u1FC8'..'\u1FCB' 
| '\u1FD8'..'\u1FDB' 
| '\u1FE8'..'\u1FEC' 
| '\u1FF8'..'\u1FFB' 
| '\u2102' 
| '\u2107' 
| '\u210B'..'\u210D' 
| '\u2110'..'\u2112' 
| '\u2115' 
| '\u2119'..'\u211D' 
| '\u2124' 
| '\u2126' 
| '\u2128' 
| '\u212A'..'\u212D' 
| '\u2130'..'\u2131' 
| '\u2133' 
| '\u213E'..'\u213F' 
| '\u2145' 
| '\uFF21'..'\uFF3A' 
| '\u010400'..'\u010425' 
| '\u01D400'..'\u01D419' 
| '\u01D434'..'\u01D44D' 
| '\u01D468'..'\u01D481' 
| '\u01D49C' 
| '\u01D49E'..'\u01D49F' 
| '\u01D4A2' 
| '\u01D4A5'..'\u01D4A6' 
| '\u01D4A9'..'\u01D4AC' 
| '\u01D4AE'..'\u01D4B5' 
| '\u01D4D0'..'\u01D4E9' 
| '\u01D504'..'\u01D505' 
| '\u01D507'..'\u01D50A' 
| '\u01D50D'..'\u01D514' 
| '\u01D516'..'\u01D51C' 
| '\u01D538'..'\u01D539' 
| '\u01D53B'..'\u01D53E' 
| '\u01D540'..'\u01D544' 
| '\u01D546' 
| '\u01D54A'..'\u01D550' 
| '\u01D56C'..'\u01D585' 
| '\u01D5A0'..'\u01D5B9' 
| '\u01D5D4'..'\u01D5ED' 
| '\u01D608'..'\u01D621' 
| '\u01D63C'..'\u01D655' 
| '\u01D670'..'\u01D689' 
| '\u01D6A8'..'\u01D6C0' 
| '\u01D6E2'..'\u01D6FA' 
| '\u01D71C'..'\u01D734' 
| '\u01D756'..'\u01D76E' 
| '\u01D790'..'\u01D7A8' 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 