[antlr-interest] Range generation in C++ mode

[Martin Probst]

Hi,

>> If I compile & run the code on windows I get an exception that a debug
>> assertion has failed in isctype.c (?) if I put UTF-8 chars into my input
>> file. Is this a known bug or problem?
>
> It sounds vaguely familiar. Is this using MSVC? What does the assertion
> say? It could be some locale related setting that may send isprint
> somewhere down a wrong path (if this comes from isprint)? A backtrace of
> the crash could also help.

Yes, it happens when compiling with MSVC 7.1 (2003). I've attached the
stack backtrace. The precise message is:
> Debug Assertion Failed!
> Program: ...
> File: isctype.c
> Line: 56
> Expression: (unsigned)(c+1) <= 256

It's really called from isprint as you can see from the backtrace.

> I also got a hack that has better UTF8 support in C++. Using a modified
> InputBuffer and another CharScanner superclass. The InputBuffer decodes
> the
> UTF8 and the CharScanner superclass can repackage things in the form you
> like (currently UTF8 stored in std::string)

I'm actually quite content with the way I handle it at the moment. The
only problem I will get is proper error reporting (with respect to column
numbers) but I'll either do this in a special error handler or with an
ugly client-side hack. Converting UTF-8 to UCS-2 or sth similar would not
actually help me as I have to compare the string values of tokens to UTF-8
strings later on.

Regards,
Martin

 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/
 
-------------- next part --------------
>	XQuery Grammar.exe!_chvalidator(int c=-33, int mask=343)  Line 56 + 0x2a	C
 	XQuery Grammar.exe!isprint(int c=-33)  Line 183 + 0x2d	C
 	XQuery Grammar.exe!antlr::charName(int ch=-33)  Line 43 + 0x9	C++
 	XQuery Grammar.exe!antlr::MismatchedCharException::getMessage()  Line 82 + 0x3f	C++
 	XQuery Grammar.exe!antlr::TokenStreamRecognitionException::TokenStreamRecognitionException(antlr::RecognitionException & re={...})  Line 23 + 0x4c	C++
 	XQuery Grammar.exe!XQueryLexer::nextToken()  Line 68 + 0xf	C++
 	XQuery Grammar.exe!antlr::TokenBuffer::fill(unsigned int amount=1)  Line 50 + 0x19	C++
 	XQuery Grammar.exe!antlr::TokenBuffer::LT(unsigned int i=1)  Line 65	C++
 	XQuery Grammar.exe!antlr::LLkParser::LT(unsigned int i=1)  Line 51 + 0x21	C++
 	XQuery Grammar.exe!XQueryParser::versionDecl()  Line 112 + 0x19	C++
 	XQuery Grammar.exe!XQueryParser::module()  Line 44	C++
 	XQuery Grammar.exe!main(int argc=2, char * * argv=0x00321b10)  Line 77	C++
 	XQuery Grammar.exe!mainCRTStartup()  Line 259 + 0x19	C
 	kernel32.dll!77e814c7()