[antlr-interest] Antlr 3 and the newline token problem
open.zone at virgin.net
Sat Nov 26 08:21:47 PST 2005
> > I can. Attempting to tack on line/col/offset counting to a
> lexer via
> > virtual method overrides for instance.
> That's basically one virtual method call (newline();) per
> line break,
If the lexer core is doing it or has specific support for detecting newlines
which isn't what you were advocating.
In any case you've omitted the per-character call for col/offset tracking.
We were discussing line/col/offset counting not just newlines.
> which is in turn one pointer de-referencing plus
> a method call per newline. E.g. with a 4000 line file you get
> 4000 pointer de-referencing ops more with a virtual function
> instead of a "normal" function.
If the lexer was built to do it properly, there would be no function calls
> I don't know what you're
> doing with the 4000 lines you have parsed in the same time,
> but are 4000 de-refs really significant compared to stepping
> through the parsing rules for 4000 lines of code and building the AST?
Lexers don't build ASTs. The per-char calls needed for line/col/offset
tracking would definitely hurt lexer performance if the counts were tacked
on via overridden methods.
More information about the antlr-interest