[antlr-interest] ST 4.0 planning

[Terence Parr]

Dear ST-o-philes and related humans,

I am starting the planning stages for ST 4.0. I begin my sabbatical  
next May, a few months after I finish this current book. I plan on  
writing software like crazy (for 15 months!). This will include  
optimizing ANTLR and hopefully converting ANTLR and ST to be ANTLR v3  
clean; no more ANTLR v2 requirement.

As part of converting ST to v3 grammars, I took a look at updating  
things.  As I looked through all the complicated code that manages  
dynamic scopes, parameters, and nested templates I realized that there  
is a lot of stuff going on in there.  ST groove organically from a  
simple string with holes in it to a sophisticated tree-based  
interpreter. Tree-based Interpreters are much more difficult to build  
then, say, a byte code interpreter. Further, debugging ST stuff is  
quite difficult because all you have are objects and you have to chase  
a lot of pointers through hash tables and so on to figure out what is  
going on.  There is no code to step through related to your templates.

I am contemplating moving to a JSP-like model where I generate Java  
(or C# or Python, ...) instead of doing an interpreter. There are a  
number of advantages:

1. In principle, we could use the rechargeable architecture pattern of  
ANTLR to generate whatever source code we want; C++ and so on. the  
only requirement would be some sort of reflection still because I  
don't want attributes to be typed in ST. That means that you'd need  
RTTI for C++, which it supposedly has now.

2.  I would suspect that the templates would go much faster when  
executing "natively" in Java.

3. You could debug templates by stepping through them just like you do  
ANTLR parsers. Templates would translate to Java methods. Groups would  
translate to objects.  Like JSP, we could automatically compile things  
in the background. This means that they would go slow the first time  
you ran the template. Also, I would have to investigate a custom class  
loader so that I could unload templates.

I'm planning on breaking with absolute backward compatibility to fix a  
number of design flaws that came about because requirements changed  
during the last eight years.

So, it is a bit premature, but I like to have things to think about  
while I'm waiting in line etc...

The idea of generating Java code is growing on me. Note that it would  
only be generating Java or stay an interpreter. I would not do both.  
Those are two totally separate products almost in terms of  
implementation.

Ter