[antlr-interest] ANTLR 3.0 lexer speed :)

Terence Parr parrt at cs.usfca.edu
Tue Nov 16 17:27:57 PST 2004


Ok, I'm starting to check out lexer speed for Java.  Comparing pascal 
lexer speed (no parsing for now).  I turn on -Xint and -Xmx200M to 
avoid compilation and GC interruption.  I load entire file into memory 
with FileReader.read(char[]).  Naturally, the new char[file.length()] 
takes a while to allocate and zero out the memory.  Ick.  memory mapped 
I/O should be faster I imagine, but "should" never seems to match up to 
my expectations with Sun's libraries. ;)

Anyhoo, with -Xint on, ANTLR 3.0 is about 6 times faster even before I 
optimize the DFAs; I "backtrack" (i.e., test chars twice) now on each 
keyword I'm pretty sure whereas the 2.0 lexer uses a hashtable for 
keywords.

I run each input file a bunch of times on each test to prevent the VM 
start up from skewing the results.

Now, allowing the on-the-fly java compiler, ANTLR 3 is about 5.5 times 
faster than ANTLR 2 so about the same, which is comforting.

I noticed, however, that if i tweaked my test file then it didn't get 
compiled dynamically and the damn thing takes LONGER than ANTLR 2.  
Ugh.  A statically-compiled language would be nice sometimes ;)

Just to give some absolute numbers, on my 1Ghz mac os x laptop with 1G 
ram, it takes 2.58s to lex a 177k pascal file 50 times with ANTLR 3; 
that's 8.85M of text throughput.  The ANTLR 2 version takes about 
14.42s.

I'm not sure Ruslan will be happy yet, but I am at this early stage of 
the game. ;)  I'm not sure how much of the difference is the fully 
buffered char stream, but whatever.  I'll take the new numbers.

Terence
--
CS Professor & Grad Director, University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Cofounder, http://www.jguru.com
Cofounder, http://www.knowspam.net enjoy email again!





 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
    antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/
 





More information about the antlr-interest mailing list