[antlr-interest] Memory management of C target

Jim Idle jimi at temporal-wave.com
Mon Jan 31 11:09:37 PST 2011


The pointer to the token (unless it is an imaginary token produced by the
parser) is in the token field of pANTLR3_COMMON_TREE. So, take the pointer
to the base tree that the reference in the tree gives you and cast the
super field in there to (pANTLR3_COMMON_TREE) and then the token field is
pANTLR3_COMMON_TOKEN. However, the getToken() method of
pANTLR3_COMMON_TREE will do that for you. Look at the methods in
antlr3commontree.c.

Remember that the tree parser only deals with pointers to the lowest basic
structure which is pANTLR3_BASE_TREE and that has a pointer 'super' to the
structure that contains it (normally pANTLR3_COMMON_TREE) which also has a
suprt pointer in case you encapsulate it further (usually too much hassle
to be worth it).

For a node with children then follow the lists recursively.

The code that produces dot files for an arbitrary tree is a good place to
look for hints as this traverses pANTLR3_BASE_TREE and looks for the text
that represents it. You will find that in antlr3basetreeadapator.c

Jim



> -----Original Message-----
> From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> bounces at antlr.org] On Behalf Of Karim Chichakly
> Sent: Monday, January 31, 2011 10:53 AM
> To: antlr-interest at antlr.org interest
> Subject: Re: [antlr-interest] Memory management of C target
>
> Hi Jim,
>
> It is clear that for tokens in the parser, you can use getStartIndex
> and getStopIndex directly to avoid using $text.  How can you do this
> for an arbitrary tree node when walking the tree?  It appears in this
> case that you also need the token stream (to ask for the token using
> get()).  Is there any way to get the token stream from the tree node or
> is there another way to get the text associated with the node?
>
> Thanks,
>
> Karim
>
>
> On Mon, Jan 31, 2011 at 12:43 PM, Jim Idle <jimi at temporal-wave.com>
> wrote:
>
> > The C target will be a lot faster than the Java target, but the
> > objects that are created are probably bigger. For v4 I plan to reduce
> that a lot.
> > It is probably better to reduce the input though. 530,000 lines of C
> > code as input seems a bit of a tall order for anything, even if you
> parse it.
> > The individual input files would be better.
> >
> > Also, I think you were using $text references in your parser and
> these
> > will create hundreds of thousands of string objects that will not be
> > released until you release the parser. To use the text of an object
> it
> > is better to get the pointer to the input from that object and use
> the
> > length (start and end pointer are stored in the object) so that you
> > make no copies or memory allocations. The $text (in the C target) is
> a
> > convenience method that is relatively slow and inefficient; it is
> just
> > there when you don't really care that much about those factors. This
> > catches so many people that I may abandon it in v4, in favor of
> > functions/macros that give you the information.
> >
> > You can also try 64bit mode, which will raise the 2GB bar.
> >
> > Jim
> >
> >
> >
> > > -----Original Message-----
> > > From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-
> > > bounces at antlr.org] On Behalf Of Marco Trudel
> > > Sent: Monday, January 31, 2011 5:37 AM
> > > To: antlr-interest at antlr.org
> > > Subject: [antlr-interest] Memory management of C target
> > >
> > > Dear all
> > >
> > > Does anyone know how the C target handles memory? I noticed that
> > > with very big input (e.g. 530.000 lines of C code) it crashes
> > > because it hits the 2gb process memory limit. Is there something I
> > > can tweak to make it work or do I have to split the input?
> > >
> > > The Java target manages to parse the input if I give the process
> 1gb.
> > > It even requires only 20 seconds.
> > > Would be great if the C target could also do that. Even better it
> > > the required time would be about half of the one of the Java target
> > > (as I'm used to when the C target can handle the input).
> > >
> > > Thanks
> > > Marco
> > >
> > > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > > Unsubscribe:
> > > http://www.antlr.org/mailman/options/antlr-interest/your-
> > > email-address
> >
> > List: http://www.antlr.org/mailman/listinfo/antlr-interest
> > Unsubscribe:
> > http://www.antlr.org/mailman/options/antlr-interest/your-email-
> address
> >
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address


More information about the antlr-interest mailing list