Even pre-2.3, you should have seen gains by adding threads, if indeed
your hardware has good concurrency.
And definitely with the changes in 2.3, you should see gains by
adding threads.
Note that as you add threads, the "sweet spot" for RAM buffer size
increases. Ie, make the RAM buffer bigger as you add more threads.
I think the only major thing that's single-threaded is flushing a new
segment to disk. Only one thread can do that, and while that thread
is doing so, other threads must wait.
Mike
Jake Mannix wrote:
------=_Part_3862_23986701.1202102642086
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
The test in which we got the 11X speedup? That was single
threaded. I
haven't yet found a way to make multithreaded (shared IndexWriter)
indexing
perform with any better speed than singlethreaded, so that code is not
enabled in our tests. Do you think that 2.3 would better take
advantage of
multiple threads / cores? If so, I could rerun it again
multithreaded and
see if that's even better...
-jake
On Feb 3, 2008 9:02 PM, ajay_garg
<[EMAIL PROTECTED]>
wrote:
Hi Jake.
Was the test conducted with a single indexing thread, or multiple
ones ?
Jake Mannix wrote:
Hello all,
I know you lucene devs did a lot of work on indexing
performance in
2.3,
and I just tested it out last thursday, so I thought I'd let you
know
how
it
fared:
On a 2.17 million document index, a recent test gave indexing
time to
be:
* lucene 2.2: 4.83 hours
* lucene 2.3: 26 minutes
About a factor of 11 speedup. Holy smokes! Great work folks.
-jake
--
View this message in context:
http://www.nabble.com/Indexing-Speed%3A-2.3-vs-2.2-%28real-world-
numbers%29-tp15257512p15262216.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
