[antlr-interest] C target memory usage

Richard Gildea rgildea at gmail.com
Sat Dec 31 06:21:01 PST 2011


Thanks, that is interesting. I didn't realise that AST construction could
double the memory usage.

On 29 December 2011 17:59, A Z <asicaddress at gmail.com> wrote:

> I ran some more experiments using valgrind to profile the heap allocations
> and I now see about 70:1 using only my modified lexer on a 64-bit system.
> Enabling parsing with AST construction roughly doubles this.
>
>
>
> On Mon, Dec 26, 2011 at 11:33 PM, Jim Idle <jimi at temporal-wave.com> wrote:
>
> > Are you sure that this is not because you are using $text? Anyway, over
> > the next few days I am going to make some changes to the runtime for
> speed
> > and memory use, that will probably help everyone.
> >
> > Jim
> >
> > > -----Original Message-----
> > > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > > bounces at antlr.org] On Behalf Of Richard Gildea
> > > Sent: Thursday, December 22, 2011 8:00 PM
> > > To: antlr-interest at antlr.org
> > > Subject: [antlr-interest] C target memory usage
> > >
> > > Hi,
> > >
> > > We have been successfully using antlr in the form of the C target for
> > > some time, however we have recently noticed that the memory consumption
> > > can be quite large - up to 150 times the size of the input file. Is
> > > this factor of
> > > ~150 to be expected, or does it indicate that we may be doing something
> > > wrong? For the vast majority of possible inputs this does not cause a
> > > problem, however some input files can be as large as 0.5 Gb, giving a
> > > peak memory usage of 75 Gb - not exactly feasible on most machines!
> > >
> > > Does anyone have any examples of using a custom lexer that provides a
> > > token buffer rather than storing all tokens in memory?
> > >
> > > Cheers,
> > >
> > > Richard
> > >
> > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> > > email-address
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-address
> >
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>


More information about the antlr-interest mailing list