[antlr-interest] C++ and Unicode

[Ric Klaren]

Hi,

On Sat, Aug 14, 2004 at 02:26:52PM +0200, Martin Probst wrote:
> I'm about to start writing a parser which has to support Unicode. I want
> to use ANTLR for this task and the output has to be Unicode. I was quite
> surprised to see that currently Unicode with C++ and ANTLR is not really
> supported. The only thing I found about it is the patch or special
> distribution (?) by Ric Klaren.
> 
> Can somebody point me to information about the current status of C++
> Unicode support in ANTLR? Is the Unicode version on this page:
> <http://wwwhome.cs.utwente.nl/~klaren/antlr/right.html> in a usable
> state or more some kind of development?

The patch on my page is a hack there's a better approach detailed a while
ago by Mark Lentczner.

See this thread:

http://groups.yahoo.com/group/antlr-interest/messages/11772

I also got another 'hack' that makes the C++ part read UTF8 and store it in
the backend in std::string but UTF8 encoded. The framework for that can be
adapted quite easily to deal with other input encodings and output
encodings.

I'm waiting for some patches Mark promised me a while back. After that I'll
release a new snapshot with 2.7.4 bugfixes, the C++ port of the
TokenStreamRewriteEngine, UnicodeCharBuffer and UnicodeCharScanner. Also a
new reference counter will be used for tokens (for starters). Character
literals are limited to \ufffe though due to the ANTLR 2 analysis engine.

Cheers,

Ric
--
-----+++++*****************************************************+++++++++-------
    ---- Ric Klaren ----- j.klaren at utwente.nl ----- +31 53 4893755  ----
-----+++++*****************************************************+++++++++-------
   Words fly like arrows
      as if we knew what was right and wrong. --- Chuang Tsu



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/