[antlr-interest] ANTLR 3.0 lexer speed :)
Terence Parr
parrt at cs.usfca.edu
Tue Nov 16 17:27:57 PST 2004
Ok, I'm starting to check out lexer speed for Java. Comparing pascal
lexer speed (no parsing for now). I turn on -Xint and -Xmx200M to
avoid compilation and GC interruption. I load entire file into memory
with FileReader.read(char[]). Naturally, the new char[file.length()]
takes a while to allocate and zero out the memory. Ick. memory mapped
I/O should be faster I imagine, but "should" never seems to match up to
my expectations with Sun's libraries. ;)
Anyhoo, with -Xint on, ANTLR 3.0 is about 6 times faster even before I
optimize the DFAs; I "backtrack" (i.e., test chars twice) now on each
keyword I'm pretty sure whereas the 2.0 lexer uses a hashtable for
keywords.
I run each input file a bunch of times on each test to prevent the VM
start up from skewing the results.
Now, allowing the on-the-fly java compiler, ANTLR 3 is about 5.5 times
faster than ANTLR 2 so about the same, which is comforting.
I noticed, however, that if i tweaked my test file then it didn't get
compiled dynamically and the damn thing takes LONGER than ANTLR 2.
Ugh. A statically-compiled language would be nice sometimes ;)
Just to give some absolute numbers, on my 1Ghz mac os x laptop with 1G
ram, it takes 2.58s to lex a 177k pascal file 50 times with ANTLR 3;
that's 8.85M of text throughput. The ANTLR 2 version takes about
14.42s.
I'm not sure Ruslan will be happy yet, but I am at this early stage of
the game. ;) I'm not sure how much of the difference is the fully
buffered char stream, but whatever. I'll take the new numbers.
Terence
--
CS Professor & Grad Director, University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Cofounder, http://www.jguru.com
Cofounder, http://www.knowspam.net enjoy email again!
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/antlr-interest/
<*> To unsubscribe from this group, send an email to:
antlr-interest-unsubscribe at yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
More information about the antlr-interest
mailing list