[antlr-interest] Exception inthread"main" java.lang.OutOfMemoryError: Java heap space

[siva.kumar at loglogic.com]

Okay, this was my mistake. I was using code that I got from some sample grammar and was using:

        testLexer lex = new testLexer(new ANTLRFileStream(args[0]));

According to the documentation for ANTLRFileStream:

  This is a char buffer stream that is loaded from a file all at once when 
  you construct the object. This looks very much like an ANTLReader or 
  ANTLRInputStream, but it's a special case since we know the exact size of 
  the object to load. We can avoid lots of data copying.

So the code was trying to load the entire file before parsing it.

Surprisingly, the almost equivalent C code, ran for some time before core-dumping:

    pANTLR3_INPUT_STREAM           input;
    ptestLexer               lex;
    pANTLR3_COMMON_TOKEN_STREAM    tokens;
    ptestParser              parser;

    input  = antlr3AsciiFileStreamNew          ((pANTLR3_UINT8)argv[1]);
    lex    = testLexerNew                (input);
    tokens = antlr3CommonTokenStreamSourceNew  (ANTLR3_SIZE_HINT, TOKENSOURCE(lex));
    parser = testParserNew               (tokens);

    parser  ->logfile(parser);


Thanks,
-Siva

-----Original Message-----
From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Siva Kumar (siva.kumar at loglogic.com)
Sent: Thursday, June 19, 2008 2:06 PM
To: Jim Idle; antlr-interest at antlr.org
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Exception inthread"main" java.lang.OutOfMemoryError: Java heap space

Thanks for your suggestion, Jim.

The only place that I use/refer to the objects is in the parser in code:

keyvaluetokens  : ID EQ STRING COMMA
                {
                        String mykey = $ID.text;
                        System.out.println("ID IN TABLE= " + $ID.text + ",VALUE=" + $STRING.text) ;
                }
                ;

(The entire grammar is at:
http://pastie.textmate.org/218178
)

Will try to switch to C and see what happens. :-)

Thanks,
-Siva

________________________________________
From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Jim Idle
Sent: Thursday, June 19, 2008 10:38 AM
To: antlr-interest at antlr.org
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] Exception in thread"main" java.lang.OutOfMemoryError: Java heap space

On Thu, 2008-06-19 at 10:30 -0700, siva.kumar at loglogic.com wrote: 

I have a simple grammar at:

http://pastie.textmate.org/218178

The LOG file that's parsed is of the form:

ABCD="....", EF="aaaa", abcd="1.2.3.4",hostname=a at b.com,

While I build and run this using antlr and java, I get the "Java heap
space" problem on a very large file (70M). I don't have a problem with
smaller files

Try using the -Xincgc option with your runtime, but are you doing anything that would cause the runtime to try and create huge strings such as the entire input lots of times in the lexer, or calling the routine that produces the dot spec for graphviz (that blows up after the number of nodes gets somewhat large. Or creating objects and  references and then never freeing up the references to these objects?

You could always switch to the C output, where you could at least find out where the memory was going to ;-)

Jim