[antlr-interest] (follow up) setting, altering text in lexer rules

Terence Parr parrt at cs.usfca.edu
Mon Jun 12 12:30:39 PDT 2006


ok, so lexing time when from 1340ms to 2095ms when I added all this  
crap in there.  That is more than 1.5x the cost in time.  I can't  
justify that 50% increase in lexer time.  Also the parser is taking  
more time...weird...a GC issue?

Wow. the

text.setLength(0);

in the nextToken() method costs 200ms out of that 2095ms.  When I  
remove all this machinery it goes back to what it was in my notes  
time-wise so it's not a fluke.  Rats!

Ok, I propose that we take a big step back and say "you can set the  
text for the token manually".  You get a setText() method and the  
auto mechanism will see your altered text if nonnull.  If you want to  
build up a token piecemeal you must do so manually.  So you'd do this:

ESC : '\\' 'n' {setText("\n");} ;

I still need to spend time inc/dec the rule level though so I know  
when to emit a token.  It seems to cost a wee bit but that is ok I  
guess.

Ter



More information about the antlr-interest mailing list