[antlr-interest] Antlr 3 and the newline token problem

[Ric Klaren]

Martin Probst wrote:
> Maybe I'm lacking imagination, but in what parsing task do you actually
> expect that counting lines/offsets is a significant performance
> bottleneck? I really can't imagine anything complex enough to justify
> using something like ANTLR but so trivial that line counting might give
> significant performance differences.
> 
> About the virtual methods: maybe some C++ crack might be able to change
> the ANTLR C++ code creation and move lots of the stuff now handled by
> virtual functions into some template magic. That would of course ruin
> compatibility with old compilers, so probably both modes should be
> available.

For the C++ support code I've been experimenting with this. It works
really great, by plugging in different Position Tracking objects you get
a customized lexer that only tracks what's needed. Even on small toy
grammars this safes quite some space. (and hopefully also overhead)

The only thing I still need to figure out is the d*mn unicode handling,
where you have issues with binary offset and logical (character) offset.
For unicode I only want to decode a character once, but that's tricky
atm with how I set up the prototype.

I'll only maintain one version e.g. a templatized version. A C++
compiler that can't deal with basic templates should go the dino way
asap. But if someone wants to maintain another version he's welcome :)

Cheers,

Ric