[antlr-interest] Lexer speed comparison

[Terence Parr]

Hi Tom,

Good questions.

Also, did you have testLiterals=false in your Lexer options?  WIth that 
as true, it does a hashtable lookup for each token.

Ter
On Feb 25, 2004, at 5:41 PM, Tom Moog wrote:

>
> a. Did you use flex -F or -f ?
>
> b. Did you try to eliminate I/O time in some way (e.g. by parsing
> from memory rather than a file) ?
>
> c. Did the java tools time require 8-bit => 16 bit conversion or
> were they 8 bit character scanners or was code conversion done
> only once before the timing loop ?
>
> d.   Did you allow "warm-up" time for the jit ?
>
> Tom
>
>
> On Wed, 25 Feb 2004, Uli Bubenheimer wrote:
>
>> I ran some tests comparing Lexers generated by different tools in
>> terms of speed. The sophistication of my Lexer specification that I
>> used as input for the Lexer generators is about that of a lexer for C.
>> The average numbers I found by repeatedly running the generated lexers
>> on a large source file were these:
>>
>> ANTLR           8300ms
>> JLex            6500ms
>> JFlex 1.4pre5   2000ms
>> Flex            1900ms
>>
>> I used current versions of all tools. For the ANTLR comparison I had
>> to adjust the Lexer specification, but tried to make it as efficient
>> as possible. For the Flex comparison, I basically called Flex's
>> yylex() function from Java via JNI. Also in the case of Flex I
>> generated fewer Java objects as output than for the others, and if I
>> take that into account, Flex's time would probably be more similar to
>> JFlex's.
>>
>> I was surprised to find that ANTLR generated the slowest lexer.
>> Naturally, when interpreting these numbers keep in mind that the
>> outcome may have been different on a different lexer specification or
>> lexer input text. On the other hand, it makes sense to me that a DFA
>> would be faster on average than a more powerful recursive-descent 
>> parser.
>>
>> Are there other performance numbers available? I have not been able to
>> dig up anything substantial. I also wonder how ANTLR-generated LL(k)
>> parsers perform compared to LALR(1) parsers.
>>
>> Uli
>>
>>
>>
>>
>>
>> Yahoo! Groups Links
>>
>>
>>
>>
>>
>
>
>
> Yahoo! Groups Links
>
>
>
>
>
>
--
Professor Comp. Sci., University of San Francisco
Creator, ANTLR Parser Generator, http://www.antlr.org
Cofounder, http://www.jguru.com
Cofounder, http://www.knowspam.net enjoy email again!
Cofounder, http://www.peerscope.com pure link sharing





 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/