[antlr-interest] C target memory usage

A Z asicaddress at gmail.com
Fri Dec 23 19:09:03 PST 2011


Hi Richard,

  I see about 140:1 for the ratio of memory use to input size on a 64-bit
system. This is after I hacked commontoken to remove most of the function
pointers, which halved the size of the tokens. I didn't investigate any
further as I recall an email thread about ANTLR 4 indicating it would use
8-byte tokens instead of the 200+ byte tokens 3.4 uses.


Looks like it may only be for C#.
http://markmail.org/message/eggfcjt3a6qdzkvc

Ad


On Fri, Dec 23, 2011 at 10:00 AM, Richard Gildea <rgildea at gmail.com> wrote:

> Hi,
>
> We have been successfully using antlr in the form of the C target for some
> time, however we have recently noticed that the memory consumption can be
> quite large - up to 150 times the size of the input file. Is this factor of
> ~150 to be expected, or does it indicate that we may be doing something
> wrong? For the vast majority of possible inputs this does not cause a
> problem, however some input files can be as large as 0.5 Gb, giving a peak
> memory usage of 75 Gb - not exactly feasible on most machines!
>
> Does anyone have any examples of using a custom lexer that provides a token
> buffer rather than storing all tokens in memory?
>
> Cheers,
>
> Richard
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe:
> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>


More information about the antlr-interest mailing list