[antlr-interest] Re : Re : reuse() methos in 3.4 C runtime/exception report

Rampon Jerome ramponjerome at yahoo.fr
Mon Nov 21 12:41:26 PST 2011



Thanks Jim for feedback.
Original run was with a full 3.4 distribution (I do not know precisely what is C library distributed in it)


I made a try with latest C runtime library I know (3c-3.4) and
this problem clearly persists. I checked compilation ok and settings properly modified in 

script to point properly on new library.

I use a 3.4 antlr version and 3c-3.4 for C library on 64bits.

If you need more details, let me know.

Jerome



________________________________
 De : Jim Idle <jimi at temporal-wave.com>
À : antlr-interest at antlr.org 
Envoyé le : Lundi 21 Novembre 2011 18h30
Objet : Re: [antlr-interest] Re : reuse() methos in 3.4 C runtime/exception report
 
I believe that is fixed in the latest source.

Jim

> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Rampon Jerome
> Sent: Monday, November 21, 2011 7:13 AM
> To: Ruslan Zasukhin; antlr-interest at antlr.org
> Subject: [antlr-interest] Re : reuse() methos in 3.4 C
> runtime/exception report
>
>
>
> Hi Jim, Ruslan,
>
>
> I experimented some issue in a similar kind of usage (iteration of
> multiple inputstream with same lexer & parser)
>
> I reset the tokenstream, lexer and parser in order specified in Jim's
> answer ...
>
> Unfortunately, for second iteration and up, some tokens in factory do
> not see their
>
> input field properly set (until pools is reallocated).
>
> Reset of the lexer calls the factory reset but do not clean up the
> input field of tokens already in the pool.
>
> Later on when generating new tokens, their are picked up first in the
> pool but setCharIndex is not NULL and input not modified.
>
> In case of exception (tokenstream) in 2nd or upper files, filename
> reported is retrieved from token->input->filename resulting in a
> confusing reporting.
>
> if not clear, let me know and I 'll try to give you more details.
> If anything wrong in my analysis, feedbacks are welcome.
>
> I did not find any simple and clean way with antlr3 API  to reset those
> input files fields for already-in-the-pool tokens
>
>
> Thanks.
> Jerome
> PS: Anyway, in term of runtime, this approach in C is not significantly
> reducing time but it
>
> makes huge differences in Java. For my grammar, with a suite of 1800
> input files, it takes around 6 mns for a JAVA 1 file at a time call vs
> seconds for a serialisation of input files
>
>
>
> ________________________________
>  De : Ruslan Zasukhin <ruslan_zasukhin at valentina-db.com>
> À : "antlr-interest at antlr.org" <antlr-interest at antlr.org> Envoyé le :
> Jeudi 17 Novembre 2011 22h24 Objet : Re: [antlr-interest] reuse()
> methos in 3.4 C runtime
>
> Hi Jim,
>
> Below are copy-pastes of my class-wrapper around ANTLR3
> Lexer/Parser/TreeParser.
> So you can see if I made some stupid mistake...
>
>
> >> On 6/24/11 7:49 PM, "Jim Idle" <jimi at temporal-wave.com> wrote:
> >>
> >> Because the documentation is not yet up to date, here is an example
> >> of reusing the allocated memory in input streams and token streams:
> >>
> >>     for (i=0; i<iterations; i++)
> >>    {
> >>         // Run the parser.
> >>         psr->start(psr);
> >>
> >>         // --------------------------------------
> >>         // Now reset everything for the next run.
> >>         // Order of calls is important.
> >>
> >>         // Input stream can now be reused
> >>         input->reuse(input, sourceCode, sourceLen, sourceName);
> >>
> >>         // Reset the common token stream so that it will reuse its
> >>resources
> >>         tstream->reset(tstream);
> >>
> >>         // Reset the lexer (new function generated by antlr now)
> >>         lxr->reset(lxr);
> >>
> >>         // Reset the parser (new function generated by antlr now)
> >>         psr->reset(psr);
> >>     }
>
>
> /**********************************************************************
> ****
> void SqlParser_v3::ResuseParserObjects(
>     const char*        inTextToParse,
>     vuint32            inLength )
> {
>     // -------------------------------
>     // TREE PARSER cannot be reused. Destroy it.
>     //
>     if( mpTreeParser )
>     {
>         mpTreeParser->free( mpTreeParser );
>         mpTreeParser = NULL;
>     }
>
>     if( mpNodes )
>     {
>         mpNodes->free( mpNodes );
>         mpNodes = NULL;
>     }
>
>
>     // -------------------------------
>     // Reuse other objects
>     //
>     mpInput->reuse(
>         mpInput,
>         (pANTLR3_UINT8) inTextToParse,
>         (ANTLR3_UINT32) inLength,
>         (pANTLR3_UINT8) "VSQL" );
>
>     mpTokenStream->reset( mpTokenStream );
>     mpLexer         ->reset( mpLexer );
>     mpParser     ->reset( mpParser );
>
>     ResetOwnData( mpParser );
> }
>
>
>
> And few other related methods ...
>
>
> /**********************************************************************
> ****
> void SqlParser_v3::Parse_UTF8(
>     I_SqlDatabaseEx*           inDatabase,
>     const char*                inCommand,
>     const char*                inCommandEnd ) {
>     argused1(inDatabase);
>
> //    COMMENT this line to force  REUSE() mode ...
> //    DestroyParserObjects();
>
>     if( mpInput )
>         ResuseParserObjects( inCommand, (inCommandEnd - inCommand) );
>     else
>         CreateParserObjects( inCommand, (inCommandEnd - inCommand) );
>
>
>     // -------------------------
>     // Parse the input expression
>     mAST = mpParser->sql( mpParser );
>
>
>     // IF PARSER have generate some errors,
>     // then we throw them as VSQL exception.
>     if( mpParser->pParser->rec->state->errorCount )
>     {
>         StToUTF16 cnv( ResultStringBuffer, pErrEnd, GetConverter_UTF8()
> );
>         throw VSQL::xVSQLException( ERR_SQL_PARSER_ERROR, cnv.c_str()
> );
>     }
> }
>
>
> /**********************************************************************
> ****
> void SqlParser_v3::CreateParserObjects(
>     const char*        inTextToParse,
>     vuint32            inLength )
> {
>     if( inTextToParse == NULL )
>         return; // all objects will be still NULLs also.
>
>     // ------------------------------
>     // Create INPUT object:
>
>     // NOTE: SQL strings do not have BOM - first few bytes, which
> define
> endian of UTF16.
>     //         So for UTF16, we must here self specify BE or LE.
>     mpInput = antlr3StringStreamNew(
>         (pANTLR3_UINT8) inTextToParse, mEncoding, (ANTLR3_UINT32)
> inLength,
> (pANTLR3_UINT8) "VSQL" );
>
>     mpInput->setUcaseLA( mpInput, ANTLR3_TRUE );
>
>
>     // ------------------------------
>     // Create LEXER v3 object:
>
>     mpLexer = SqlParser_v3LexerNew( mpInput );
>     mpTokenStream = antlr3CommonTokenStreamSourceNew( ANTLR3_SIZE_HINT,
> TOKENSOURCE( mpLexer ) );
>
>
>     // ------------------------------
>     // Create PARSER v3 object:
>
>     mpParser = SqlParser_v3ParserNew( mpTokenStream );  // is generated
> by
> ANTLR3
>     mpParser->mDoAllCommands = mDoAllCommandsInitial;
>
>     ResetOwnData( mpParser );
>
>
> /**********************************************************************
> ****
> void SqlParser_v3::DestroyParserObjects( void )
> {
>     // REVERSE ORDER to construction:
>
>     if( mpTreeParser )
>     {
>         mpTreeParser->free( mpTreeParser );
>         mpTreeParser = NULL;
>     }
>
>     if( mpNodes )
>     {
>         mpNodes->free( mpNodes );
>         mpNodes = NULL;
>     }
>
>     if( mpParser )
>     {
>         mpParser->mpStartPositions = NULL;
>
>         mpParser->free( mpParser );
>         mpParser = NULL;
>     }
>
>     if( mpTokenStream )
>     {
>         mpTokenStream->free( mpTokenStream );
>         mpTokenStream = NULL;
>     }
>
>     if( mpLexer )
>     {
>         mpLexer->free( mpLexer );
>         mpLexer = NULL;
>     }
>
>     if( mpInput )
>     {
>         mpInput->close( mpInput );
>         mpInput = NULL;
>     }
> }
>
>
>
>
>
>
> --
> Best regards,
>
> Ruslan Zasukhin
> VP Engineering and New Technology
> Paradigma Software, Inc
>
> Valentina - Joining Worlds of Information
> http://www.paradigmasoft.com
>
> [I feel the need: the need for speed]
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address


More information about the antlr-interest mailing list