[antlr-interest] [C target] [3.1.1] Memory consumption goes through the roof with rewrite rules

Jim Idle jimi at temporal-wave.com
Tue Jan 20 09:29:40 PST 2009


Sven Van Echelpoel wrote:

Yes, this is fixed (or should be) in the current source code snapshot. 
Perhaps it isn't and that is why the other poster was creating so many 
tree nodes. That's why I wanted to seen an example. Changing dupTree 
isn't the answer it is higher of the chain where dupTree should not be 
being called. Perhaps I never checked in the fix. I am making some time 
this week for this and other runtime investigation..

Jim
> Hi,
>
> after passing a piece of non-trivial source code through the parser of
> the language we created, memory consumption went through the roof (it
> immediately consumed the full 3GB of memory on my 64 bit Ubuntu box) and
> this for a source file of a measly 14K characters. Using a debugger and
> a profiler I immediately came to the same conclusion as this poster:
> http://preview.tinyurl.com/a94qgn. Recursive invocation of rewrite rules
> results in addChild() and dupTree() being invoked millions of times.
>
> While his example was slightly contrived, because of the problem domain
> our language is highly recursive in nature. And although the source file
> was slightly bigger than a toy sample, it is by no means what we intend
> to use it for.
>
> So, desperately wanting to get to the bottom of this I used Jim Idle's
> response to the aforementioned post (emphasis mine):
>
> <quote>
> That looks like a bug - in fact, I think I remember saying to myself 
> "now I have put in factories for everything, I should *not call dup on 
> the trees*! However, I also remember doing that, so perhaps it is 
> something to do with your grammar) Sorry, but I still won't be able to 
> look in to this until the weekend :-( 
> </quote>
>
> to determine that I could maybe short-circuit the code in dupTree() in
> 'antlr3basetree.c'.
>
> Here's what I did:
>
> static void    *
> dupTree		(pANTLR3_BASE_TREE tree)
> {
>   /* Fixes memory explosion when using rewrite rules */
>   return tree; 
>   /* Rest of code removed */
> }
>
> in other words, instead of duplicating the tree, I just return it. This
> seems to solve the problem. Memory consumption is back to normal and the
> code seems to run OK.
>
> The questions that I have are these:
>
> 1) Is this workaround valid or are there some unforeseen bodies still
> left in the closet?
> 2) If this is a bug, as was indicated in the response to the OP, will it
> get fixed in one of the following released of ANTLR?
>
> Thanks for the input,
>
> Sven
>
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>   



More information about the antlr-interest mailing list