[antlr-interest] Re: Lexer speed comparison

[Uli Bubenheimer]

Thanks for the feedback.

a. I did not use flex -F or -f. I used it out of the box, like "flex
myspec.lex"
b. I parsed from memory.
c. I used as input whatever the tool asked for and was easy to
implement. ANTLR, JLex, and JFlex each got a Reader (reading from a
char array in memory), Flex got a character array (via
yy_scan_bytes()). In the ANTLR, JLex, and JFlex specifications I
specified 8-bit character sets.
d. All initialization (calling constructors, initializing Flex, etc.)
was done outside the timing loop. Inside the loop, I only made the
calls to retrieve the next token. I ran each test a number of times
with similar results on each run.

I did not set testLiterals=false. My grammar has keywords. I guess I
could alternatively recognize them by setting the lookahead to
something like k=10. But would such a large lookahead really speed
things up?

Unfortunately I am not able to publish the proprietary specification I
used (it's a real-world grammar that my company uses for its
products). But as I indicated, it is not terribly sophisticated. I
would expect similar results, for example, with a lexer spec. for (a
subset of) C or Java. Perhaps someone would be interested in running
such tests?

Uli

--- In antlr-interest at yahoogroups.com, Terence Parr <parrt at c...>
wrote:
> Hi Tom,
> 
> Good questions.
> 
> Also, did you have testLiterals=false in your Lexer options?  WIth
that 
> as true, it does a hashtable lookup for each token.
> 
> Ter
> On Feb 25, 2004, at 5:41 PM, Tom Moog wrote:
> 
> >
> > a. Did you use flex -F or -f ?
> >
> > b. Did you try to eliminate I/O time in some way (e.g. by parsing
> > from memory rather than a file) ?
> >
> > c. Did the java tools time require 8-bit => 16 bit conversion or
> > were they 8 bit character scanners or was code conversion done
> > only once before the timing loop ?
> >
> > d.   Did you allow "warm-up" time for the jit ?
> >
> > Tom



 
Yahoo! Groups Links

<*> To visit your group on the web, go to:
     http://groups.yahoo.com/group/antlr-interest/

<*> To unsubscribe from this group, send an email to:
     antlr-interest-unsubscribe at yahoogroups.com

<*> Your use of Yahoo! Groups is subject to:
     http://docs.yahoo.com/info/terms/