[stringtemplate-interest] Generating big documents

[John Snyders]

I would have thought this would work but now I can see it doesn't.

The problem is in ASTExpr.applyListOfAlternatingTemplates. It is trying to
gather up the results of the first template. I think the reason it is doing
this is because it is possible to nest the templates. The example from the
documentation is
<ul>
$names:{<b>$it$</b>}:{<li>$it$</li>}$
</ul>First all the names are iterated over and a result list of string
templates with arguments initialized is returned. Then the list of string
templates is evaluated. In the first loop the values are it=0, it=1, it=2...
in the second loop the values are it=<b>0</b>, it=<b>1</b>, ...
I was surprised to see it done this way. I think it is very inefficient and
clearly will not work for what you are trying to do. It would be nice if it
could go depth first rather than breadth first. If not at least notice that
there is no nested template to iterate in your case.

Can someone comment on why nested iterations are handled this way?

Note that $data$ does not have the same problem. The values are continually
written to the DummyWriter. But this is not very helpful. A named template
will also not work ex: $data:line()$

I think your use case is an interesting and important one. Not sure if this
issue can be resolved and there may be more issues beyond that.

There are problems with using iterators directly. There are times when they
need to be (even if ST isn't currently doing it) turned into a collection
(or iterable). The reason is that iterators can't be used more than once.
This is why Java 5 added the itertor interface to support the foreach
construct. It needed a way to get an iterator multiple times when a foreach
loop was nested. But this is not the key issue in what you are doing. You
could change your example to be an infinite collection and run into the same
problem.


-John

  -----Original Message-----
  From: stringtemplate-interest-bounces at antlr.org
[mailto:stringtemplate-interest-bounces at antlr.org]On Behalf Of Adam Bennett
  Sent: Wednesday, December 27, 2006 6:03 PM
  To: stringtemplate-interest at antlr.org
  Subject: Re: [stringtemplate-interest] Generating big documents


  I think this gets me close, Nate.  I did not realize that StringTemplate
treats iterators as multi-valued attributes.  Nice.  I could create an
Iterator to wrap a JDBC ResultSet.  One problem however: I think
StringTemplate is evaluating the iterator all at once (as opposed to
iteratively).  For instance, this code causes an OutOfMemoryError:

  public class Play
  {
    public static void main(String[] args)
    {
      StringTemplate template = new StringTemplate("$data:{$i$ ==
$it$\n}$");
      template.setAttribute("data", new InfiniteItr());

      try
      {
        template.write(new DummyWriter());
      }
      catch (java.io.IOException ex)
      {
        ex.printStackTrace();
      }
     }

    /**an infinite sequence of integers*/
    static class InfiniteItr implements Iterator<Integer>
    {
      private int i = 0;
      public boolean hasNext() {return true;}
      public Integer next(){return new Integer(i++); }
      public void remove(){}
    }

    /**a StringTemplateWriter that discards all output*/
    static public class DummyWriter
      implements StringTemplateWriter
    {
      public void popAnchorPoint() {}
      public String popIndentation() {return "";}
      public void pushAnchorPoint(){}
      public void pushIndentation(String string){}
      public void setLineWidth(int _int){}
      public int write(String string)
        throws IOException
      {
        return string.length();
      }
      public int write(String string, String string1)
        throws IOException
      {
        return string.length();
      }
      public int writeSeparator(String string)
        throws IOException
      {
        return string.length();
      }
      public int writeWrapSeparator(String string)
        throws IOException
      {
        return string.length();
      }
    }
  }




----------------------------------------------------------------------------
    From: Nate [mailto:misc at n4te.com]
    To: Adam Bennett [mailto:adamb at videx.com]
    Cc: stringtemplate-interest at antlr.org
    Sent: Wed, 27 Dec 2006 11:04:39 -0800
    Subject: Re: [stringtemplate-interest] Generating big documents

    You could try something like this...

    ArrayList list = new ArrayList();
    list.add("10");
    list.add("20");
    list.add("30");
    StringTemplate template = new StringTemplate("$data:{$i$ == $it$\n}$");
    template.setAttribute("data", list.iterator());
    System.out.println(template.toString());

    Outputs...

    1 == 10
    2 == 20
    3 == 30

    -Nate


    Adam Bennett wrote:
    > Our web application has the ability to generate reports as HTML or XML
    > documents. These documents are potentially very large (20MB with some
    > of our test data). Early on we had problems with the server running
    > out of memory while generating these because all the data was brought
    > into memory from the database. Our solution was to stream the results
    > from the database while generating the document.
    >
    > Can StringTemplate be used to generate such documents? I cannot see
    > how to avoid bringing all the data into memory because StringTemplate
    > requires all data to be passed as an attribute.


  > ------------------------------------------------------------------------
    >
    > _______________________________________________
    > stringtemplate-interest mailing list
    > stringtemplate-interest at antlr.org
    > http://www.antlr.org:8080/mailman/listinfo/stringtemplate-interest
    >



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.antlr.org:8080/pipermail/stringtemplate-interest/attachments/20061227/a5f8a77c/attachment.html