[antlr-interest] Antlr lexer vs. flex lexer speed (C++)

[Alex Sedow]

Ric,

1. The result will be better if you replace antlr or flex lexer with
re2c-based lexer.

  Generated lexers:
  antlr 2.x - procedure-driven lexer (slowest)
  flex - table-driven
  re2c - case-statement-driven (fastest)

  In any case 5282790 bytes / 11.780s = 0.45 Mb/sec - is slow.

2. Whether ANTLR 3.x will use std::string, or will replace it with special
string class?

Alex.

> Ran a small test for a question that entered via private mail but the
> result is interesting enough to forward to the list as well...
>
> A comparison of the flex lexer java parser versus the complete antlr java
> parser.
>
> Input a subset of the java files from antlr:
>
> lines: 124509 words:  621304 bytes: 5282790
>
> With antlr's lexer:
>
> real    0m21.702s
> user    0m21.255s
> sys     0m0.211s
>
> With flex as lexer:
>
> real    0m12.008s
> user    0m11.780s
> sys     0m0.156s
>
> These are the 'average-ish' values after running the parsers a few times
> should be dependable enough to give an indication. Note that the flex
lexer
> is not as good as the java one: it lacks lexer states to handle escaped
> strings properly, but if I recall right the overhead of adding those will
> be neglectible. (anyone better into flex can shed some light on that, I'm
a
> bit on the lazy side to look up how to add them to the flexLexer example)
>
> Cheers,
>
> Ric



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/