[antlr-interest] Antlr v4 - C++ target

[Loring Craymer]

I think a bit of caution is in order.  The average text length of a token is 5 characters/bytes or less; pointers are now 8 bytes.  Tokens encapsulate state information as well as text--input source, character index, line number, token type, C++ vtable pointers, and so forth--and are necessarily much larger than the text they represent.  Data items are aligned on word (8 byte) boundaries, which adds even more expansion.  It is possible to compress data structures by removing alignment constraints, converting pointers to offsets (instead of an end of text pointer, keep a byte or short integer index), and some other tricks; these all cost in terms of performance.  Further, there is the issue of minimum allocation size; allocated memory chunks contain hidden pointers and (depending on the compiler) are likely to be allocated in chunks that are multiples of 128 bytes or larger.  Shrinking ANTLR data structures, which are already near-minimal in terms
 of preserved information, is not easy to do.  That is especially true for C++; the reason that Jim Idle did a C target for ANTLR 3 in the first place was to reduce the footprint of the ANTLR 2 C++ target.

So--despite Sam's intentions, I would not expect the ANTLR 4 C++ memory footprint to be smaller than the ANTLR 3 C footprint.  If Sam does manage to produce a small footprint target, feel free to applaud wildly.  Just don't expect miracles until they are made real.

--Loring



>________________________________
> From: Richard Gildea <rgildea at gmail.com>
>To: Sam Harwell <sam at tunnelvisionlabs.com> 
>Cc: antlr-interest at antlr.org 
>Sent: Tuesday, January 10, 2012 5:44 PM
>Subject: Re: [antlr-interest] Antlr v4 - C++ target
> 
>Hi Sam,
>
>Thanks for the info!
>
>Richard
>
>On 10 January 2012 16:22, Sam Harwell <sam at tunnelvisionlabs.com> wrote:
>
>> Hi Richard,
>>
>> I wouldn't expect a C++ version of ANTLR 4 within the next 12 months. The
>> Java target and ANTLRWorks 2 are both still pre-alpha. I'm not planning to
>> start the C# target until a release candidate of v4 is available, and the
>> C#
>> target will involve ports of the runtime, the tool, and potentially
>> integration of some ANTLRWorks 2 features into Visual Studio. After that is
>> working I'll start the C++ port.
>>
>> I'm confident that under standard usage the C++ port will reduce memory to
>> about 10-15% of what 3.4 currently requires. Using templates and a custom
>> token type, it should be possible to cut that down to <5% of what 3.4
>> requires.
>>
>> --
>> Sam Harwell
>> Owner, Lead Developer
>> http://tunnelvisionlabs.com
>>
>>
>> -----Original Message-----
>> From: Richard Gildea [mailto:rgildea at gmail.com]
>> Sent: Tuesday, January 10, 2012 5:19 PM
>> To: antlr-interest at antlr.org
>> Subject: [antlr-interest] Antlr v4 - C++ target
>>
>> Hi,
>>
>> These are mainly questions for Terrence/Sam Harwell (who I believe will be
>> writing the C++ target):
>>
>> 1) Is there even a vague estimate of when a likely C++ target is likely to
>> be available for v4 (even an alpha/beta version)? Is it likely to be on the
>> scale of e.g. 2-3 month, 4-6 months, 6 months-1 year, 1 year+?
>>
>> 2) Is the C++ target expected to have a much lower memory footprint than
>> the
>> current v3.4 C target?
>>
>> After recently realising that the v3.4 C target has a largish memory
>> footprint (see my recent emails to the list), there has been some pressure
>> among some of my co-developers to move to a handwritten parser over ANTLR
>> (especially from those who were suspicious of using ANTLR from the start)
>> in
>> order to reduce the memory usage. Our parser is required to read files as
>> large as 0.5 Gb, hence the pressure to keep memory usage as low as
>> possible.
>>
>> It sounds like v4 will likely be a big improvement over v3.4, and I think
>> knowing the answers to the above questions could influence whether or not
>> we
>> move away from ANTLR with our parser.
>>
>> Cheers,
>>
>> Richard
>>
>> List: http://www.antlr.org/mailman/listinfo/antlr-interest
>> Unsubscribe:
>> http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>>
>>
>
>List: http://www.antlr.org/mailman/listinfo/antlr-interest
>Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address
>
>
>