[antlr-interest] Recovering white space in V3.0
parrt at cs.usfca.edu
Sat Jun 4 14:50:57 PDT 2005
On Jun 4, 2005, at 2:24 PM, Scott Stanchfield wrote:
> (I'll be nice and trim the "to" line for ya)
> Am I reading correctly that all tokens are held on to for the
> duration of
> the parse? If so, this could cause problems for parsing very long
> files (esp
> XML) that don't need remembered state...
My default CommonTokenStream does buffer everything...for parsing 20G
files, you would use a simple one that didn't buffer. I am building
stuff in general to work for the common case not the extremes,
leaving the ability to handle the extremes.
More information about the antlr-interest