[antlr-interest] Recovering white space in V3.0

[Andy Tripp]

On this issue of "common" vs. "extreme" and whether to buffer all tokens 
or not...

I think the main thing is that you want ANTLR to provide all the 
functionality that
most people might want, and let people override (or turn off) stuff they 
don't want.
Some people may want tokens buffered  and others won't. Better to 
provide the buffering
code and let people turn it off, rather than not buffering everything 
and having people who need
it have to write code to do it.

In other words, have CommonTokenStream do buffering and then maybe provide a
alternative LeanTokenStream that doesn't. But don't just provide 
LeanTokenStream,
because then people will have to write their own buffering code.

Andy