[antlr-interest] [C target] [3.1.1] Memory consumption goes through the roof with rewrite rules

[Sven Van Echelpoel]

Hi,

after passing a piece of non-trivial source code through the parser of
the language we created, memory consumption went through the roof (it
immediately consumed the full 3GB of memory on my 64 bit Ubuntu box) and
this for a source file of a measly 14K characters. Using a debugger and
a profiler I immediately came to the same conclusion as this poster:
http://preview.tinyurl.com/a94qgn. Recursive invocation of rewrite rules
results in addChild() and dupTree() being invoked millions of times.

While his example was slightly contrived, because of the problem domain
our language is highly recursive in nature. And although the source file
was slightly bigger than a toy sample, it is by no means what we intend
to use it for.

So, desperately wanting to get to the bottom of this I used Jim Idle's
response to the aforementioned post (emphasis mine):

<quote>
That looks like a bug - in fact, I think I remember saying to myself 
"now I have put in factories for everything, I should *not call dup on 
the trees*! However, I also remember doing that, so perhaps it is 
something to do with your grammar) Sorry, but I still won't be able to 
look in to this until the weekend :-( 
</quote>

to determine that I could maybe short-circuit the code in dupTree() in
'antlr3basetree.c'.

Here's what I did:

static void    *
dupTree		(pANTLR3_BASE_TREE tree)
{
  /* Fixes memory explosion when using rewrite rules */
  return tree; 
  /* Rest of code removed */
}

in other words, instead of duplicating the tree, I just return it. This
seems to solve the problem. Memory consumption is back to normal and the
code seems to run OK.

The questions that I have are these:

1) Is this workaround valid or are there some unforeseen bodies still
left in the closet?
2) If this is a bug, as was indicated in the response to the OP, will it
get fixed in one of the following released of ANTLR?

Thanks for the input,

Sven