[antlr-interest] Antlr 3 and the newline token problem

[Micheal J]

> > I can. Attempting to tack on line/col/offset counting to a 
> lexer via 
> > virtual method overrides for instance.
> 
> That's basically one virtual method call (newline();) per 
> line break,

If the lexer core is doing it or has specific support for detecting newlines
which isn't what you were advocating.

In any case you've omitted the per-character call for col/offset tracking.
We were discussing line/col/offset counting not just newlines.

> which is in turn one pointer de-referencing plus 
> a method call per newline. E.g. with a 4000 line file you get 
> 4000 pointer de-referencing ops more with a virtual function 
> instead of a "normal" function.

If the lexer was built to do it properly, there would be no function calls
at all.

> I don't know what you're 
> doing with the 4000 lines you have parsed in the same time, 
> but are 4000 de-refs really significant compared to stepping 
> through the parsing rules for 4000 lines of code and building the AST?

Lexers don't build ASTs. The per-char calls needed for line/col/offset
tracking would definitely hurt lexer performance if the counts were tacked
on via overridden methods.

Micheal