[antlr-interest] String size limit and how to free memory real-time?

Rick Morgan r.morgan at verizonbusiness.com
Fri Nov 10 08:35:59 PST 2006


regarding my own questions:
1) there doesn't seem to be a built-in limit since, in the c++ target code
anyway, there is a circular queue being used which is implemented with a
vector. The vector is self-expanding for you non-c++ types (which includes
me for the most part, but I'm learning). There is a trip point in the code
set at 5000 that causes the vector to shuffle things around somehow, but I
didn't study it hard enough to figure out the purpose. I don't think my 7800
character (or bigger) tokens are impeded by that 5000 number, not really
sure on that point.

Delving further into my problem I found that the stack overflow occurs when
the parser destructor is called.  It attempts to free the AST which in my
case is a degenerate list of some sort, containing 25800 nodes.  The
compiler apparently generates code that frees these up using recursive stack
frames which eventually blows the stack.  I'm looking at how to implement a
non-recursive version of it but c++ isn't my strong point and I can't really
see how the list is being destructed in the first place (recursion is just
my theory at the moment), so don't see where to intervene.

2) I found a removeChildren method in the BaseAST class (a base class of my
parser I believe), but when I attempt to call it, I get a no-such-method
error at compile time.  I probably need to cast the result of my #myVar
reference to BaseAST, but I got way off in the weeds with number 1) above,
so haven't got back to this.  Any suggestions on how to free up the AST
real-time would be helpful, as well as warnings about corrupting it all.

thanks,
Rick

> Message: 2
> Date: Wed, 08 Nov 2006 16:48:41 -0700
> From: Rick Morgan <r.morgan at verizonbusiness.com>
> Subject: [antlr-interest] String size limit and how to free memory
> 	real-time?
> To: antlr-interest at antlr.org
> Message-ID: <000201c70390$6ef23710$ea1a9283 at mcilink.com>
> Content-Type: text/plain; charset=us-ascii
> 
> Hi,
> 
> 2 things:
> 
> 1)is there a configuration variable that controls how large a token can be
> when ANTLR is parsing/lexing?  I am getting the following error when my
> program shuts down (seems to run fine until the EOF token is matched):
> 
> Pid 4507 received a SIGSEGV for stack growth failure.
> Possible causes: insufficient memory or swap space,
> or stack size exceeded maxssiz.
> Segmentation fault
> 
> The parser itself seems to run fine, it's the return from my main program
> that induces the error, but none of this happened until I changed my
> grammar
> to handle some rather large "tokens" (7800 or more chars.)  The file I'm
> parsing is only 86k, so I'm surprised I'm running into this.
> 
> Is there some limit I need to change somewhere?
> 
> 2) I think if I could free up parts of the AST tree that's been built,
> parts
> I no longer need once my action routines have fired, then I could reduce
> the
> memory utilization. Right now it shoots up to about 140Meg on the above
> 86k
> file before exiting the program and that's mostly the parser and its AST
> because I stubbed out all the application code.  The ASTFactory.cpp code
> I'm
> using from the distribution doesn't have any type of prune operations that
> I
> can see.
> 
> Any suggestions on how to free up some memory real-time? Do I need to
> create
> my own AST factory for this purpose?
> 
> thanks,
> Rick





More information about the antlr-interest mailing list