[antlr-interest] [Antlr + ST] Huge output string - OutOfMemoryError

[Hieu Phung]

Hi all,

I am writing a program using Antlr and StringTemplate to translate an input
file to an XML file. Input is small but output is huge: 720KB input file
will generate about 100MB output file. When my input reached 7608KB, I got
OOM error

time taken in second: 18 - file size in KB: 4804
time taken in second: 20 - file size in KB: 5205
time taken in second: 20 - file size in KB: 5605
time taken in second: 21 - file size in KB: 6006
time taken in second: 23 - file size in KB: 6406
time taken in second: 25 - file size in KB: 6806
time taken in second: 31 - file size in KB: 7207

at size 7608KB:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
        at java.lang.StringBuffer.append(StringBuffer.java:320)
        at java.io.StringWriter.write(StringWriter.java:60)
        at
org.antlr.stringtemplate.AutoIndentWriter.write(AutoIndentWriter.java:172)
        at
org.antlr.stringtemplate.language.ASTExpr.writePOJO(ASTExpr.java:702)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:683)
        at
org.antlr.stringtemplate.language.ASTExpr.writeAttribute(ASTExpr.java:660)
        at
org.antlr.stringtemplate.language.ActionEvaluator.action(ActionEvaluator.java:86)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:149)
        at
org.antlr.stringtemplate.StringTemplate.write(StringTemplate.java:705)
        at
org.antlr.stringtemplate.language.ASTExpr.writeTemplate(ASTExpr.java:750)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:680)
        at
org.antlr.stringtemplate.language.ASTExpr.writeIterableValue(ASTExpr.java:792)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:682)
        at
org.antlr.stringtemplate.language.ASTExpr.writeAttribute(ASTExpr.java:660)
        at
org.antlr.stringtemplate.language.ActionEvaluator.action(ActionEvaluator.java:86)
        at org.antlr.stringtemplate.language.ASTExpr.write(ASTExpr.java:149)
        at
org.antlr.stringtemplate.StringTemplate.write(StringTemplate.java:705)
        at
org.antlr.stringtemplate.StringTemplate.toString(StringTemplate.java:1670)
        at
org.antlr.stringtemplate.StringTemplate.toString(StringTemplate.java:1661)
        at FFMParser.file(FFMParser.java:127)
        at CSVTest.main(CSVTest.java:33)

Is there any suggestion so that program can generate 1GB file?

1)       Having multiple templates, each template generates small string,
then write the small strings to file.
My input has format: ((Part1) (Part2)+ (Part3)). For each part2, there is an
xml element in the output. I can have 3 templates here. In Antlr action, I
will write each part to file when parsing is done for the part.

2)       Any template engine that could write to file directly, instead of
returning a string...? (This might not be proper to discussed here :D)

I hope problem is stated clearly. Please let me know if you need more
information. Any suggestion is much appreciated. Thank you.

Regards,

Helen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org/pipermail/antlr-interest/attachments/20091016/10495bd3/attachment.html