[antlr-interest] [Bug] Antlr C code memory leak in Antlr 3.1.1

Jim Idle jimi at temporal-wave.com
Wed Nov 12 07:58:15 PST 2008 

On Wed, 2008-11-12 at 09:42 +0000, merry_winfly at hotmail.com wrote:
> 
> Hi Jim,
>  
> 
> I have test two g file, and one exist memory leak and anther no memory leak issue, more importantly, two files only one line is different.
>  
> And this two g file all come from PLSQL3.g(only change a little).
>  
> And I give you and memory leak information and all code in attachment.
>  
> I think this is a bug of antlr3.1.1 C.
>  
> Any way, I should fix the problem for this is a important demo with urgent schedule, and hope get help from you.
>  

OK - as I tried to explain to you, your parser produces an AST tree
output. However you are just calling the parser and throwing away the
return value of the parser rule. Copy the tree parsing example code and
create a node stream from the return value. You should then close this
node stream down as in the examples. Also check the strings you are
passing in to the parser - at first glance you are creating them
dynamically and not destroying them.

If you do not wish to use an AST, then take out the output=AST option
and the AST rewriting rules.

Finally you are saying: k=* and have turned on backtracking mode
globally. This parser will therefore be pretty inefficient and it will
not be able to pin down syntax errors too well for you. Turn off
backtracking mode and fix up the ambiguities that backtrack is hiding.

I will verify my code exception by fixing your code later today all
being well.

Jim



> Thanks a lot.
>  
> Best Regards
> Winfly Lin
> 
> 
>  
> 
> 
> 
> From: merry_winfly at hotmail.com
> To: jimi at temporal-wave.com
> CC: antlr-interest at antlr.org
> Subject: RE: [antlr-interest] Antlr C code memory leak in Antlr 3.1.1
> Date: Wed, 12 Nov 2008 02:21:36 +0000
> 
> 
> 
> Use the attachment to see the source code for the display issue (I can't send the all code for large size, any more information pls tell me directly).
>  
> Thanks 
> Winfly Lin
> > 
> >> From: merry_winfly at hotmail.com
> >> To: jimi at temporal-wave.com
> >> Date: Wed, 12 Nov 2008 02:10:08 +0000
> >> CC: antlr-interest at antlr.org
> >> Subject: Re: [antlr-interest] Antlr C code memory leak in Antlr 3.1.1
> >> 
> >> 
> >> Hi Jim
> >> 
> >> =>
> >> pANTLR3_INPUT_STREAM lInput=NULL;
> >> pANTLR3_COMMON_TOKEN_STREAM lPredigestTokenStream=NULL;
> >> pPLSQLpredigestParser lPredigestParser=NULL;
> >> pPLSQLpredigestLexer lPredigestLexer=NULL;
> >> lInput = antlr3NewAsciiStringInPlaceStream((pANTLR3_UINT8)SQL.c_str(), (ANTLR3_UINT64)SQL.size(), NULL);
> >> if(!lInput)
> >> {
> >> cout<<" input error" 
> >> lPredigestLexer = PLSQLpredigestLexerNew(lInput);
> >> if(!lPredigestLexer)
> >> {
> >> cout<<" lPredigestLexer error" 
> >> lPredigestParser->free(lPredigestParser);
> >> lPredigestTokenStream->free(lPredigestTokenStream);
> >> lPredigestLexer->free(lPredigestLexer);
> >> lInput->close(lInput);
> >> lPredigestParser=NULL;
> >> lPredigestTokenStream=NULL;
> >> lPredigestLexer=NULL;
> >> lInput=NULL;
> >> 
> >> =>This is my all source code, and memory leak exist in PLSQLpredigestParser_start_rule_return psr = lPredigestParser->start_rule(lPredigestParser);
> >> 
> >> 
> >> Actually, I have two type PLSQL's lexer and parser code from two type PLSQL g file. 
> >> 
> >> But only this file (PLSQLPredigest.g,I send to you before)exist memory leak (the source PLSQL.g is from website, and no memory leak issue,PLSQLpredigest is predigested from PLSQL.g) , and I seems to follow the "treeparser" sample to code (because I don't have any walker to new a node "antlr3CommonTreeNodeStreamNewTree"). 
> >> 
> >> Thank Jim.
> >> 
> >> Best Regards
> >> Winfly Lin
> >> 
> >>> Subject: Re: [antlr-interest] Antlr C code memory leak in Antlr 3.1.1
> >>> From: jimi at temporal-wave.com
> >>> To: merry_winfly at hotmail.com
> >>> CC: antlr-interest at antlr.org
> >>> Date: Tue, 11 Nov 2008 08:52:30 -0800
> >>> 
> >>> On Tue, 2008-11-11 at 13:39 +0000, merry_winfly at hotmail.com wrote:
> >>>> 
> >>>> ==>
> >>>> Actully, I find when (lPredigestParser->free(lPredigestParser);) the adapter and vector will be freed by this free action, but at last some tokenstream didn't be freed, I don't know if my source code issue or antlr's bug.
> >>>> 
> >>>> And any one know this, pls freely tell me the reason.
> >>> 
> >>> Did you call free on the token stream itself? I think that this is your
> >>> issue. There is a treenode stream that you must free too. Check against
> >>> that example tree parsers, but I will look at your issue today.
> >>> 
> >>> Jim
> >>> 
> >>>> 
> >>>> Thanks a lot.
> >>>> 
> >>>> Best Regards
> >>>> Winfly Lin
> >>>> 
> >>>> 
> >>>> 
> >>>> From: merry_winfly at hotmail.com
> >>>> To: antlr-interest at antlr.org
> >>>> Date: Mon, 10 Nov 2008 13:21:04 +0000
> >>>> Subject: [antlr-interest] Antlr C code memory leak in Antlr 3.1.1
> >>>> 
> >>>> 
> >>>> 
> >>>> Hi All,
> >>>> 
> >>>> When I use the C code from antlr generator with predigest PLSQL parser , I find it exist obvious memory leak issue.
> >>>> 
> >>>> I use valgrind to check and the result is:
> >>>> 
> >>>> 
> >>>> valgrind --leak-check=full ./treeMemory
> >>>> 
> >>>> ==9541== Memcheck, a memory error detector.
> >>>> 
> >>>> ==9541== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
> >>>> 
> >>>> ==9541== Using LibVEX rev 1854, a library for dynamic binary translation.
> >>>> 
> >>>> ==9541== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
> >>>> 
> >>>> ==9541== Using valgrind-3.3.1, a dynamic binary instrumentation framework.
> >>>> 
> >>>> ==9541== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
> >>>> 
> >>>> ==9541== For more details, rerun with: -v
> >>>> 
> >>>> ==9541==
> >>>> ==9541==
> >>>> ==9541== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 17 from 1)
> >>>> ==9541== malloc/free: in use at exit: 8,008 bytes in 98 blocks.
> >>>> ==9541== malloc/free: 8,847 allocs, 8,749 frees, 2,202,767 bytes allocated.
> >>>> ==954 1== For counts of detected errors, rerun with: -v
> >>>> ==9541== searching for pointers to 98 not-freed blocks.
> >>>> ==9541== checked 149,260 bytes.
> >>>> ==9541==
> >>>> ==9541== 8,008 (7,560 direct, 448 indirect) bytes in 90 blocks are definitely lost in loss record 3 of 3
> >>>> ==9541== at 0x40057C8: malloc (vg_replace_malloc.c:207)
> >>>> ==9541== by 0x8055BFA: antlr3RewriteRuleElementStreamNewAE (antlr3rewritestreams.c:94)
> >>>> ==9541== by 0x8055DA0: antlr3RewriteRuleTOKENStreamNewAE (antlr3rewritestreams.c:192)
> >>>> ==9541== by 0x80B8125: select_list (PLSQLpredigestParser.c:51593)
> >>>> ==9541== by 0x80B4E0F: select_expression (PLSQLpredigestParser.c:50283)
> >>>> ==9541== by 0x80B3F4D: select_statement (PLSQLpredigestParser.c:49890)
> >>>> ==9541== by 0x810C0F5: insert_command (PLSQLpredigestParser.c:83912)
> >>>> ==9541== by 0x80B1E12: sql_command (PLSQLpredigest Parser.c:49133)
> >>>> ==9541== by 0x80B13BE: sql_statement (PLSQLpredigestParser.c:48855)
> >>>> ==9541== by 0x8083185: statement (PLSQLpredigestParser.c:31331)
> >>>> ==9541== by 0x808110E: seq_of_statements (PLSQLpredigestParser.c:30542)
> >>>> ==9541== by 0x8077D5C: start_rule (PLSQLpredigestParser.c:26944)
> >>>> ==9541==
> >>>> ==9541== LEAK SUMMARY:
> >>>> ==9541== definitely lost: 7,560 bytes in 90 blocks.
> >>>> ==9541== indirectly lost: 448 bytes in 8 blocks.
> >>>> ==9541== possibly lost: 0 bytes in 0 blocks.
> >>>> ==9541== still reachable: 0 bytes in 0 blocks.
> >>>> ==9541== suppressed: 0 bytes in 0 blocks.
> >>>> 
> >>>> Source Code:
> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>> string SQL="insert into T K(a1,a2,a3) select a,b,c from t2 where a>9 union all select e,f,g from t5 where e>'winfly' ;";
> >>>> 
> >>>> pANTLR3_INPUT_STREAM lInput=NULL;
> >>>> pANTLR3_COMMON_TOKEN_STREAM lPredigestTokenStream=NULL;
> >>>> pPLSQLpredigestParser lPredigestParser=NULL;
> >>>> pPLSQLpredigestLexer lPredigestLexer=NULL;
> >>>> lInput = antlr3NewAsciiStringInPlaceStream((pANTLR3_UINT8)SQL.c_str(), (ANTLR3_UINT64)SQL.size(), NULL);
> >>>> if(!lInput)
> >>>> {
> >>>> cout<<" input error"> 
> >>>> lPredigestLexer = PLSQLpredigestLexerNew(lInput);
> >>>> if(!lPredigestLexer)
> >>>> {
> >>>> cout<<" lPredigestLexer error"> 
> >>>> lPredigestParser->free(lPredigestParser);
> >>>> lPredigestTokenStream->free(lPredigestTokenStream);
> >>>> lPredigestLexer->free(lPredigestLexer);
> >>>> lInput->close(lInput);
> >>>> 
> >>>> 
> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
> >>>> Actually the memory leak exist PLSQLpredigestParser_start_rule_return psr = lPredigestParser->start_rule(lPredigestParser);
> >>>> 
> >>>> I don't know why so simple code cause so obvious memory leak
> >>>> 
> >>>> 
> >>>> Could you help me to solve this? (I'm hurry for this code, thanks a lot!)
> >>>> 
> >>>> I list the information and attachment includes all code about it (C file ):
> >>>> 
> >>>> Platform: RedHat Enterprise 5 Server, virtual machine.
> >>>> GCC version: 4.1.1
> >>>> Antlr Version: 3.1.1
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> I don't know if it is enough information to trace this problem, but if you want to know any more information, pls freely contact me with this E-mail (merry_winfly at hotmail.com).
> >>>> 
> >>>> 
> >>>> Thanks a lot.
> >>>> 
> >>>> 
> >>>> Best Regards
> >>>> Winfly Lin
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> Get news, entertainment and everything you care about at Live.com. Check it out!
> >>>> _________________________________________________________________
> >>>> Connect to the next generation of MSN Messenger 
> >>>> http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline
> >>>> 
> >>>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> >>>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> >>>> 
> >>> 
> >> _________________________________________________________________
> >> Connect to the next generation of MSN Messenger 
> >> http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline
> >> 
> >> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> >> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> >> 
> > _________________________________________________________________
> > Discover the new Windows Vista
> > http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
> 
> 
> 
> 
> 
> Invite your mail contacts to join your friends list with Windows Live Spaces. It's easy! Try it!
> _________________________________________________________________
> Explore the seven wonders of the world
> http://search.msn.com/results.aspx?q=7+wonders+world&mkt=en-US&form=QBRE

More information about the antlr-interest mailing list