[antlr-interest] [Antlr + ST] Huge output string - OutOfMemoryError

[Jim Idle]

Run Java with   - Xmx1500M

 

Or some other option that increases available memory size. However this could point to something that can be better organized in terms of recursion or whatever.

 

You may also consider running Java with a better garbage collector such as:

 

-Xincgc

 

JIm

 

 

From: antlr-interest-bounces at antlr.org [mailto:antlr-interest-bounces at antlr.org] On Behalf Of Hieu Phung
Sent: Friday, October 16, 2009 4:03 PM
To: antlr-interest at antlr.org
Subject: [antlr-interest] [Antlr + ST] Huge output string - OutOfMemoryError

 

Hi all,

I am writing a program using Antlr and StringTemplate to translate an input file to an XML file. Input is small but output is huge: 720KB input file will generate about 100MB output file. When my input reached 7608KB, I got OOM error

time taken in second: 18 - file size in KB: 4804
time taken in second: 20 - file size in KB: 5205
time taken in second: 20 - file size in KB: 5605
time taken in second: 21 - file size in KB: 6006
time taken in second: 23 - file size in KB: 6406
time taken in second: 25 - file size in KB: 6806
time taken in second: 31 - file size in KB: 7207

at size 7608KB:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
        at java.lang.StringBuffer.append(StringBuffer.java:320)
        at java.io.StringWriter.write(StringWriter.java:60)
        at org.antlr.stringtemplate.AutoIndentWriter.write(AutoIndentWriter.java:172)
        at org.antlr.stringtemplate.language.ASTExpr.writePOJO(ASTExpr.java:702)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:683)
        at org.antlr.stringtemplate.language.ASTExpr.writeAttribute(ASTExpr.java:660)
        at org.antlr.stringtemplate.language.ActionEvaluator.action(ActionEvaluator.java:86)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:149)
        at org.antlr.stringtemplate.StringTemplate.write(StringTemplate.java:705)
        at org.antlr.stringtemplate.language.ASTExpr.writeTemplate(ASTExpr.java:750)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:680)
        at org.antlr.stringtemplate.language.ASTExpr.writeIterableValue(ASTExpr.java:792)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:682)
        at org.antlr.stringtemplate.language.ASTExpr.writeAttribute(ASTExpr.java:660)
        at org.antlr.stringtemplate.language.ActionEvaluator.action(ActionEvaluator.java:86)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:149)
        at org.antlr.stringtemplate.StringTemplate.write(StringTemplate.java:705)
        at org.antlr.stringtemplate.StringTemplate.toString(StringTemplate.java:1670)
        at org.antlr.stringtemplate.StringTemplate.toString(StringTemplate.java:1661)
        at FFMParser.file(FFMParser.java:127)
        at CSVTest.main(CSVTest.java:33)
       
Is there any suggestion so that program can generate 1GB file?

1)       Having multiple templates, each template generates small string, then write the small strings to file.
My input has format: ((Part1) (Part2)+ (Part3)). For each part2, there is an xml element in the output. I can have 3 templates here. In Antlr action, I will write each part to file when parsing is done for the part.

2)       Any template engine that could write to file directly, instead of returning a string...? (This might not be proper to discussed here :D)

I hope problem is stated clearly. Please let me know if you need more information. Any suggestion is much appreciated. Thank you.

Regards,

Helen

 

 

 

 

 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091018/14b5a6fb/attachment.html