[antlr-interest] multi-core usage

[Edwards, Waverly]

Thanks all for your insights.  It was very interesting.


W.

-----Original Message-----
From: Sam Harwell [mailto:sharwell at pixelminegames.com] 
Sent: Thursday, June 12, 2008 6:06 PM
To: Andy Tripp; Edwards, Waverly
Cc: antlr-interest at antlr.org
Subject: RE: [antlr-interest] multi-core usage

I'm using a combined grammar with AST output paired with a tree walker
grammar on the CSharp2 target. I spawn 5 threads on a quad-core box and
the parse uses about 65% of the overall processor power while it parses
~2500 files in ~5 seconds.

Breaking the parse into multiple threads provided about a 3:1 speed
increase. Multi-threaded parsing isn't the difficult part; it's
combining the results from the individual threads into one place that
can cause problems if you aren't careful.

Sam

-----Original Message-----
From: antlr-interest-bounces at antlr.org
[mailto:antlr-interest-bounces at antlr.org] On Behalf Of Andy Tripp
Sent: Thursday, June 12, 2008 4:50 PM
To: Edwards, Waverly
Cc: antlr-interest at antlr.org
Subject: Re: [antlr-interest] multi-core usage

I have a mechanism where I spawn multiple threads to parse multiple
files.
It speeds things up quite a bit, mostly because doing it single-threaded
is I/O bound. With the multiple threads, file reading and parsing can
happen in parallel.

As for multiple cores, I don't know how much multithreading will help.
>From what I've read, it's not easy to get Java to utilize multiple
CPUs.
I would think you'd have to have incredibly huge input for ANTLR to use
a lot of CPU to parse it. Anyway, it's pretty easy to try out
multithreading in Java, so just try it and see what happens.



Edwards, Waverly wrote:
> 
> I have curiosity question.  Can ANTLR take advantage of multi-core 
> processors.
> The only place where I can think it would be possible would be in the 
> lexer where you might be able to apply one file per core to be 
> tokenized.  Has anyone tried?
> Again, just curious.
> 
> 
> W.
>