[antlr-interest] Stream processing

[Florian Weimer]

* Micheal J.:

>> Right now, the generated parser seems to load the whole 
>> document into memory, in some kind of parsed form.  Loading 
>> the source document is not a big issue for me (it's just 
>> about 3 MB in size), but the whole parser data definitely 
>> takes too much memory (close to 1 GiB).
>
> What was the target language?.

Java.

> How did you measure this memory usage?. 

Garbage collector statistics.

Part of the problem is that CommonTokenStream builds an ArrayList of
all tokens.  I'm building my own token stream to address that.  I'm
not yet sure if this will solve the whole problem.