Re: indexing performance 6.6 vs 7.1

2018-01-31 Thread Rob Audenaerde
e a transaction log in parallel to > > indexing, > > >> so they commit very seldom. If the system crashes, the changes are > > replayed > > >> from tranlog since last commit. > > >> > > >> Uwe > > >> > > >>

Re: indexing performance 6.6 vs 7.1

2018-01-31 Thread Adrien Grand
gt; >> > >> - > >> Uwe Schindler > >> Achterdiek 19, D-28357 Bremen > >> http://www.thetaphi.de > >> eMail: u...@thetaphi.de > >> > >> > -Original Message- > >> > From: Rob Audenaerde [mailto:rob.audenae...@gmail.c

Re: indexing performance 6.6 vs 7.1

2018-01-31 Thread Rob Audenaerde
>> > -Original Message- >> > From: Rob Audenaerde [mailto:rob.audenae...@gmail.com] >> > Sent: Monday, January 29, 2018 11:29 AM >> > To: java-user@lucene.apache.org >> > Subject: Re: indexing performance 6.6 vs 7.1 >> > >> >

Re: indexing performance 6.6 vs 7.1

2018-01-29 Thread Rob Audenaerde
we > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Rob Audenaerde [mailto:rob.audenae...@gmail.com] > > Sent: Monday, January 29, 2018 11:29 AM > > To

RE: indexing performance 6.6 vs 7.1

2018-01-29 Thread Uwe Schindler
28357 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Rob Audenaerde [mailto:rob.audenae...@gmail.com] > Sent: Monday, January 29, 2018 11:29 AM > To: java-user@lucene.apache.org > Subject: Re: indexing performance 6.6 vs 7.1 > > H

Re: indexing performance 6.6 vs 7.1

2018-01-29 Thread Rob Audenaerde
Hi all, Some follow up (sorry for the delay). We built a benchmark in our application, and profiled it (on a smallish data set). What we currently see in the profiler is that in Lucene 7.1 the calls to `commit()` take much longer. The self-time committing in 6.6: 3,215 ms The self-time committin

Re: indexing performance 6.6 vs 7.1

2018-01-18 Thread Erick Erickson
Robert: Ah, right. I keep confusing my gmail lists "lucene dev" and "lucene list" Siiih. On Thu, Jan 18, 2018 at 9:18 AM, Adrien Grand wrote: > If you have sparse data, I would have expected index time to *decrease*, > not increase. > > Can you enable the IW info stream and share

Re: indexing performance 6.6 vs 7.1

2018-01-18 Thread Adrien Grand
If you have sparse data, I would have expected index time to *decrease*, not increase. Can you enable the IW info stream and share flush + merge times to see where indexing time goes? If you can run with a profiler, this might also give useful information. Le jeu. 18 janv. 2018 à 11:23, Rob Aude

Re: indexing performance 6.6 vs 7.1

2018-01-18 Thread Robert Muir
Erick I don't think solr was mentioned here. On Thu, Jan 18, 2018 at 8:03 AM, Erick Erickson wrote: > My first question is always "are you running the Solr CPUs flat out?". > My guess in this case is that the indexing client is the same and the > problem is in Solr, but it's worth checking whethe

Re: indexing performance 6.6 vs 7.1

2018-01-18 Thread Erick Erickson
My first question is always "are you running the Solr CPUs flat out?". My guess in this case is that the indexing client is the same and the problem is in Solr, but it's worth checking whether the clients are just somehow not delivering docs as fast as they were before. My suspicion is that the in

indexing performance 6.6 vs 7.1

2018-01-18 Thread Rob Audenaerde
Hi all, We recently upgraded from Lucene 6.6 to 7.1. We see a significant drop in indexing performace. We have a-typical use of Lucene, as we (also) index some database tables and add all the values as AssociatedFacetFields as well. This allows us to create pivot tables on search results really