[antlr-interest] Re: Problems with Unicode support in ANTLR

Matthew Ford Matthew.Ford at forward.com.au
Thu May 16 20:29:46 PDT 2002


This approach would not work for me as I need

IDENT
 options {testLiterals=true;
     paraphrase = "an identifier";}
 : ('a'..'z'|'_'|'$'|'\u0080'..'\uFFFE')
('a'..'z'|'_'|'0'..'9'|'$'|'\u0080'..'\uFFFE')*
 ;

So rather then sub-blocks, what I need is an efficient compression method to
store these bitsets in the Antlr.
matthew

----- Original Message -----
From: "micheal_jor" <open.zone at virgin.net>
To: <antlr-interest at yahoogroups.com>
Sent: Friday, May 17, 2002 1:10 PM
Subject: [antlr-interest] Re: Problems with Unicode support in ANTLR


>
> > Perhaps I am missing the thrust of this exchange, but
> > my requirement for Unicode in Antlr is to support an English
> command set and
> > foreign language (initially Japanese)
> > variable names.
> > This talk of character blocks seems too restrictive.
> > I need full Unicode support to cover the possible variable names.
>
> ANTLR allows you to set the charVocabulary to include almost all
> Unicode character values (the exceptions being \uFFFF - used for EOF -
>  and perhaps \u0000..\u0002).
>
> The discussion is about making ANTLR smarter so that it understands
> standard character groupings so that ANTLR .g files may be more
> compact (and understandable). There might also be a performance
> benefit (or deficit).
>
> This is very important IMO. I have included part of the definition
> required for a sample Unicode category if done manually in an ANTLR
> definition file. It is auto-generated by a tool I wrote and may
> contain errors ;-)
>
> I believe it illustrates the value of these proposals quite nicely.
>
> Micheal
>
>
> ==========================================================
>
> Unicode Category or Class: Lu
>
> | '\u0041'..'\u005A'
> | '\u00C0'..'\u00D6'
> | '\u00D8'..'\u00DE'
> | '\u0100'
> | '\u0102'
> | '\u0104'
> | '\u0106'
> | '\u0108'
> | '\u010A'
> | '\u010C'
> | '\u010E'
> | '\u0110'
> | '\u0112'
> | '\u0114'
> | '\u0116'
> | '\u0118'
> | '\u011A'
> | '\u011C'
> | '\u011E'
> | '\u0120'
> | '\u0122'
> | '\u0124'
> | '\u0126'
> | '\u0128'
> | '\u012A'
> | '\u012C'
> | '\u012E'
> | '\u0130'
> | '\u0132'
> | '\u0134'
> | '\u0136'
> | '\u0139'
> | '\u013B'
> | '\u013D'
> | '\u013F'
> | '\u0141'
> | '\u0143'
> | '\u0145'
> | '\u0147'
> | '\u014A'
> | '\u014C'
> | '\u014E'
> | '\u0150'
> | '\u0152'
> | '\u0154'
> | '\u0156'
> | '\u0158'
> | '\u015A'
> | '\u015C'
> | '\u015E'
> | '\u0160'
> | '\u0162'
> | '\u0164'
> | '\u0166'
> | '\u0168'
> | '\u016A'
> | '\u016C'
> | '\u016E'
> | '\u0170'
> | '\u0172'
> | '\u0174'
> | '\u0176'
> | '\u0178'..'\u0179'
> | '\u017B'
> | '\u017D'
> | '\u0181'..'\u0182'
> | '\u0184'
> | '\u0186'..'\u0187'
> | '\u0189'..'\u018B'
> | '\u018E'..'\u0191'
> | '\u0193'..'\u0194'
> | '\u0196'..'\u0198'
> | '\u019C'..'\u019D'
> | '\u019F'..'\u01A0'
> | '\u01A2'
> | '\u01A4'
> | '\u01A6'..'\u01A7'
> | '\u01A9'
> | '\u01AC'
> | '\u01AE'..'\u01AF'
> | '\u01B1'..'\u01B3'
> | '\u01B5'
> | '\u01B7'..'\u01B8'
> | '\u01BC'
> | '\u01C4'
> | '\u01C7'
> | '\u01CA'
> | '\u01CD'
> | '\u01CF'
> | '\u01D1'
> | '\u01D3'
> | '\u01D5'
> | '\u01D7'
> | '\u01D9'
> | '\u01DB'
> | '\u01DE'
> | '\u01E0'
> | '\u01E2'
> | '\u01E4'
> | '\u01E6'
> | '\u01E8'
> | '\u01EA'
> | '\u01EC'
> | '\u01EE'
> | '\u01F1'
> | '\u01F4'
> | '\u01F6'..'\u01F8'
> | '\u01FA'
> | '\u01FC'
> | '\u01FE'
> | '\u0200'
> | '\u0202'
> | '\u0204'
> | '\u0206'
> | '\u0208'
> | '\u020A'
> | '\u020C'
> | '\u020E'
> | '\u0210'
> | '\u0212'
> | '\u0214'
> | '\u0216'
> | '\u0218'
> | '\u021A'
> | '\u021C'
> | '\u021E'
> | '\u0220'
> | '\u0222'
> | '\u0224'
> | '\u0226'
> | '\u0228'
> | '\u022A'
> | '\u022C'
> | '\u022E'
> | '\u0230'
> | '\u0232'
> | '\u0386'
> | '\u0388'..'\u038A'
> | '\u038C'
> | '\u038E'..'\u038F'
> | '\u0391'..'\u03A1'
> | '\u03A3'..'\u03AB'
> | '\u03D2'..'\u03D4'
> | '\u03D8'
> | '\u03DA'
> | '\u03DC'
> | '\u03DE'
> | '\u03E0'
> | '\u03E2'
> | '\u03E4'
> | '\u03E6'
> | '\u03E8'
> | '\u03EA'
> | '\u03EC'
> | '\u03EE'
> | '\u03F4'
> | '\u0400'..'\u042F'
> | '\u0460'
> | '\u0462'
> | '\u0464'
> | '\u0466'
> | '\u0468'
> | '\u046A'
> | '\u046C'
> | '\u046E'
> | '\u0470'
> | '\u0472'
> | '\u0474'
> | '\u0476'
> | '\u0478'
> | '\u047A'
> | '\u047C'
> | '\u047E'
> | '\u0480'
> | '\u048A'
> | '\u048C'
> | '\u048E'
> | '\u0490'
> | '\u0492'
> | '\u0494'
> | '\u0496'
> | '\u0498'
> | '\u049A'
> | '\u049C'
> | '\u049E'
> | '\u04A0'
> | '\u04A2'
> | '\u04A4'
> | '\u04A6'
> | '\u04A8'
> | '\u04AA'
> | '\u04AC'
> | '\u04AE'
> | '\u04B0'
> | '\u04B2'
> | '\u04B4'
> | '\u04B6'
> | '\u04B8'
> | '\u04BA'
> | '\u04BC'
> | '\u04BE'
> | '\u04C0'..'\u04C1'
> | '\u04C3'
> | '\u04C5'
> | '\u04C7'
> | '\u04C9'
> | '\u04CB'
> | '\u04CD'
> | '\u04D0'
> | '\u04D2'
> | '\u04D4'
> | '\u04D6'
> | '\u04D8'
> | '\u04DA'
> | '\u04DC'
> | '\u04DE'
> | '\u04E0'
> | '\u04E2'
> | '\u04E4'
> | '\u04E6'
> | '\u04E8'
> | '\u04EA'
> | '\u04EC'
> | '\u04EE'
> | '\u04F0'
> | '\u04F2'
> | '\u04F4'
> | '\u04F8'
> | '\u0500'
> | '\u0502'
> | '\u0504'
> | '\u0506'
> | '\u0508'
> | '\u050A'
> | '\u050C'
> | '\u050E'
> | '\u0531'..'\u0556'
> | '\u10A0'..'\u10C5'
> | '\u1E00'
> | '\u1E02'
> | '\u1E04'
> | '\u1E06'
> | '\u1E08'
> | '\u1E0A'
> | '\u1E0C'
> | '\u1E0E'
> | '\u1E10'
> | '\u1E12'
> | '\u1E14'
> | '\u1E16'
> | '\u1E18'
> | '\u1E1A'
> | '\u1E1C'
> | '\u1E1E'
> | '\u1E20'
> | '\u1E22'
> | '\u1E24'
> | '\u1E26'
> | '\u1E28'
> | '\u1E2A'
> | '\u1E2C'
> | '\u1E2E'
> | '\u1E30'
> | '\u1E32'
> | '\u1E34'
> | '\u1E36'
> | '\u1E38'
> | '\u1E3A'
> | '\u1E3C'
> | '\u1E3E'
> | '\u1E40'
> | '\u1E42'
> | '\u1E44'
> | '\u1E46'
> | '\u1E48'
> | '\u1E4A'
> | '\u1E4C'
> | '\u1E4E'
> | '\u1E50'
> | '\u1E52'
> | '\u1E54'
> | '\u1E56'
> | '\u1E58'
> | '\u1E5A'
> | '\u1E5C'
> | '\u1E5E'
> | '\u1E60'
> | '\u1E62'
> | '\u1E64'
> | '\u1E66'
> | '\u1E68'
> | '\u1E6A'
> | '\u1E6C'
> | '\u1E6E'
> | '\u1E70'
> | '\u1E72'
> | '\u1E74'
> | '\u1E76'
> | '\u1E78'
> | '\u1E7A'
> | '\u1E7C'
> | '\u1E7E'
> | '\u1E80'
> | '\u1E82'
> | '\u1E84'
> | '\u1E86'
> | '\u1E88'
> | '\u1E8A'
> | '\u1E8C'
> | '\u1E8E'
> | '\u1E90'
> | '\u1E92'
> | '\u1E94'
> | '\u1EA0'
> | '\u1EA2'
> | '\u1EA4'
> | '\u1EA6'
> | '\u1EA8'
> | '\u1EAA'
> | '\u1EAC'
> | '\u1EAE'
> | '\u1EB0'
> | '\u1EB2'
> | '\u1EB4'
> | '\u1EB6'
> | '\u1EB8'
> | '\u1EBA'
> | '\u1EBC'
> | '\u1EBE'
> | '\u1EC0'
> | '\u1EC2'
> | '\u1EC4'
> | '\u1EC6'
> | '\u1EC8'
> | '\u1ECA'
> | '\u1ECC'
> | '\u1ECE'
> | '\u1ED0'
> | '\u1ED2'
> | '\u1ED4'
> | '\u1ED6'
> | '\u1ED8'
> | '\u1EDA'
> | '\u1EDC'
> | '\u1EDE'
> | '\u1EE0'
> | '\u1EE2'
> | '\u1EE4'
> | '\u1EE6'
> | '\u1EE8'
> | '\u1EEA'
> | '\u1EEC'
> | '\u1EEE'
> | '\u1EF0'
> | '\u1EF2'
> | '\u1EF4'
> | '\u1EF6'
> | '\u1EF8'
> | '\u1F08'..'\u1F0F'
> | '\u1F18'..'\u1F1D'
> | '\u1F28'..'\u1F2F'
> | '\u1F38'..'\u1F3F'
> | '\u1F48'..'\u1F4D'
> | '\u1F59'
> | '\u1F5B'
> | '\u1F5D'
> | '\u1F5F'
> | '\u1F68'..'\u1F6F'
> | '\u1FB8'..'\u1FBB'
> | '\u1FC8'..'\u1FCB'
> | '\u1FD8'..'\u1FDB'
> | '\u1FE8'..'\u1FEC'
> | '\u1FF8'..'\u1FFB'
> | '\u2102'
> | '\u2107'
> | '\u210B'..'\u210D'
> | '\u2110'..'\u2112'
> | '\u2115'
> | '\u2119'..'\u211D'
> | '\u2124'
> | '\u2126'
> | '\u2128'
> | '\u212A'..'\u212D'
> | '\u2130'..'\u2131'
> | '\u2133'
> | '\u213E'..'\u213F'
> | '\u2145'
> | '\uFF21'..'\uFF3A'
> | '\u010400'..'\u010425'
> | '\u01D400'..'\u01D419'
> | '\u01D434'..'\u01D44D'
> | '\u01D468'..'\u01D481'
> | '\u01D49C'
> | '\u01D49E'..'\u01D49F'
> | '\u01D4A2'
> | '\u01D4A5'..'\u01D4A6'
> | '\u01D4A9'..'\u01D4AC'
> | '\u01D4AE'..'\u01D4B5'
> | '\u01D4D0'..'\u01D4E9'
> | '\u01D504'..'\u01D505'
> | '\u01D507'..'\u01D50A'
> | '\u01D50D'..'\u01D514'
> | '\u01D516'..'\u01D51C'
> | '\u01D538'..'\u01D539'
> | '\u01D53B'..'\u01D53E'
> | '\u01D540'..'\u01D544'
> | '\u01D546'
> | '\u01D54A'..'\u01D550'
> | '\u01D56C'..'\u01D585'
> | '\u01D5A0'..'\u01D5B9'
> | '\u01D5D4'..'\u01D5ED'
> | '\u01D608'..'\u01D621'
> | '\u01D63C'..'\u01D655'
> | '\u01D670'..'\u01D689'
> | '\u01D6A8'..'\u01D6C0'
> | '\u01D6E2'..'\u01D6FA'
> | '\u01D71C'..'\u01D734'
> | '\u01D756'..'\u01D76E'
> | '\u01D790'..'\u01D7A8'
>
>
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>
>


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



More information about the antlr-interest mailing list