[antlr-interest] Memory management of C target

[Marco Trudel]

On 02.02.2011 18:01, Jim Idle wrote:
> I think that the fix is in there, which means that your input is too big
> to build the tree in the way it is being done. Write an input stream
> wrapper that splits the input by just returning EOF at the split point
> then resets to the next unit.

Actually I already rewrote my splitter. Now I get almost exactly the 
same performance for parsing the files splitted and as a whole.
Also the overall memory consumption is much better this way since I 
actually build an AST in another framework. I only use antlr to parse 
the input.
I just wanted to compare the performance of different approaches.
I now also tried to use the start and end pointers to the input instead 
of using $text. Doesn't make a difference in memory consumption or 
speed. So I stick with the convenient $text.

> How are you ending up with 640,000 lines of C input?

I'm doing some program analysis and preprocess the C programms with CIL 
[1]. vim and libgsl for instance then are about half a million lines of 
code.


Thanks for your time
Marco

[1] http://hal.cs.berkeley.edu/cil/


> Jim
>
>
>> -----Original Message-----
>> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
>> bounces at antlr.org] On Behalf Of Marco Trudel
>> Sent: Wednesday, February 02, 2011 8:20 AM
>> To: antlr-interest at antlr.org
>> Subject: Re: [antlr-interest] Memory management of C target
>>
>> Dear Jim
>>
>> On 01.02.2011 18:15, Jim Idle wrote:
>>> <snip>
>>>
>>> I think
>>> that in 3.3 I have fixed a bug that was not releasing memory
>>> references when building a tree until the tree was freed. Try making
>> a
>>> version that does not build a tree and see how it differs.
>>>
>>> <snip>
>>
>> Ok, made a version of my grammar that doesn't build a tree:
>>
>> libantlr3c-3.2, no tree output: works, uses about 600mb memory
>> libantlr3c-3.2, tree output: crash, wants more than 2gb
>>
>> libantlr3c-3.3-SNAPSHOT, no tree output: works, uses about 630mb memory
>> libantlr3c-3.3-SNAPSHOT, tree output: crash, wants more than 2gb
>>
>> The only changes to the grammar where removing/adding "output = AST;".
>> Is your fix already in the current SNAPSHOT?
>>
>>
>> Thanks
>> Marco
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
>> email-address
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address