[antlr-interest] Recovering white space in V3.0

Terence Parr parrt at cs.usfca.edu
Sat Jun 4 14:50:57 PDT 2005


On Jun 4, 2005, at 2:24 PM, Scott Stanchfield wrote:

> (I'll be nice and trim the "to" line for ya)
>
> Am I reading correctly that all tokens are held on to for the  
> duration of
> the parse? If so, this could cause problems for parsing very long  
> files (esp
> XML) that don't need remembered state...

My default CommonTokenStream does buffer everything...for parsing 20G  
files, you would use a simple one that didn't buffer.  I am building  
stuff in general to work for the common case not the extremes,  
leaving the ability to handle the extremes.

Ter



More information about the antlr-interest mailing list