Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-07 Thread Peter K
Hi Uwe, > Die, Maven, die :-) Well, I for myself have a love-hate-relationship to maven: its simple and works nice for deps management. also others can set it up quickly and IDE support is nice. But sometimes it does a bit too much (unexpected ;)) or is too complicated to customize. > (I assum

RE: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-07 Thread Uwe Schindler
Hi, > > I mean my benchmarks show up > > to 300% improvement with 4.x versus older versions so something is > > weird ie. non-realistic here or there is a bug so lets figure this > > out. Can you profile you app and see if you find something suspicious? > > I'll try now and report back. > > It s

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-07 Thread Peter K
> I mean my benchmarks show up > to 300% improvement with 4.x versus older versions so something is > weird ie. non-realistic here or there is a bug so lets figure this > out. Can you profile you app and see if you find something suspicious? > I'll try now and report back. It seems to be largely

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-05 Thread Peter K
Hi Simon, answers below. >> It does not seem to be an 'IO related issue' because using RAMDirectory >> results in the same times. >> And indexing via Luc4 with only one thread shouldn't be slower than 3.5 (?) > it could be since we use a different term dictionary impl which is > more expensive in

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-05 Thread Simon Willnauer
hey peter, On Wed, Jan 4, 2012 at 12:52 AM, Peter K wrote: > Thanks Simon for you answer! > >> as far as I can see you are comparing apples and pears. > > When excluding the waiting time I also get the slight but reproducable > difference**. The times for waitForGeneration are nearly the same > (

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-03 Thread Peter K
Thanks Simon for you answer! > as far as I can see you are comparing apples and pears. When excluding the waiting time I also get the slight but reproducable difference**. The times for waitForGeneration are nearly the same (~2sec). Also when I commit instead waitForGeneration it is no difference

Re: Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-03 Thread Simon Willnauer
hey Peter, as far as I can see you are comparing apples and pears. Your comparison is waiting for merges to finish and if you are using multiple threads lucene 4.0 will flush more segments to disk than 3.5 so what you are seeing is likely a merge that is still trying to merge small segments. can y

Comparing Indexing Speed of Lucene 3.5 and 4.0

2012-01-03 Thread Peter K
Hi, I recently switched an experimental project from Lucene 3.5 to 4.0 from 6th Dec 2011 and my indexing time increased by nearly 20% on my local machine*. It seems to me that two simple StringField's could cause this slow down: Field uIdField = new Field("_uid", "" + id, StringField.TYPE_STORED);