[antlr-interest] Re: feature request: Token.getOffset()

[cj_daly]

Well, thanks for all the suggestions, guys.

I think my application is actually somewhat similar to what Tom
Dudziak described in that I want to parse a file in several passes.  I
need the absolute offsets because the input data will be in a big
string (or stream) so I can seek to a given point to extract a
sub-section to parse (I would have marked the section boundary offsets
in an initial pass).

Also the editor I'm working with (the Eclipse IDE TextEditor) wants me
to give offset/length instead of line/column for things like
highlighting and selecting sections of text.  The only thing I would
need line/column for is parsing files that aren't open in the editor -
and I don't need that yet but probably will at some point.

Anyway, I'm still skeptical that counting offsets deep in the lexer
would affect performance much, but until I can come up with some test
cases that quantify the different approaches discussed here I'll keep
my mouth shut. :-)

Plus I've got a more bothersome problem with the interface of
RecognitionException.  See my next post on that....

Chris


--- In antlr-interest at yahoogroups.com, "lgcraymer" <lgc at m...> wrote:
> --- In antlr-interest at yahoogroups.com, Ric Klaren <klaren at c...> wrote:
> ..
> > (/grants loring permission to throw something at him) Hmmm so we
can get up
> > to a 20% gain in our lexers by kicking out the column stuff ;) ?
Or move
> > the checking of tabs out of the CharScanner::consume method and
handle them
> > similarly to newline by explicit calls in the lexer?
> 
> The 20% would require heavy optimization of the lexer--right now,
ANTLR lexers are grossly inefficient.  Handling tab() like newline(), 
> though, would be a good idea.  line/column information is generally
useful (it's hard to imagine a production batch translator that didn't 
> need the line/column error reporting for text processing.  Ter's
working on doing a few things that should significantly improve ANTLR 
> lexer performance.
> 
> --Loring
>   
> > Cheers,
> > 
> > Ric
> > --
> >
-----+++++*****************************************************+++++++++-------
> >     ---- Ric Klaren ----- j.klaren at u... ----- +31 53 4893722  ----
> >
-----+++++*****************************************************+++++++++-------
> >    Words fly like arrows
> >       as if we knew what was right and wrong. --- Chuang Tsu


 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/