Re: Help with huge index

2018-03-04 Thread Michael Sokolov
I wonder if you might not get better performance in a case like this if you were ok taking your index off line, disabling merges, performing deletions and only then enabling merges? This could be done on a copy of the index if updates can be turned off or held in a queue, so that queries could stil

Re: Help with huge index

2018-02-28 Thread Stuart Goldberg
Thanks so much. I actually found that my purging routine finished after about 35 minutes which is really acceptable given that this routine is supposed to run during the overnight period. On Feb 28, 2018 8:34 PM, "Adrien Grand" wrote: > Thanks. Deleting lots of documents can indeed trigger a lot

Re: Help with huge index

2018-02-28 Thread Adrien Grand
Thanks. Deleting lots of documents can indeed trigger a lot of work in the Lucene side. First Lucene likely needs to rewrite the live docs of all your segments and then this might trigger significant merging activity due to the fact that Lucene tries to keep the number of deleted docs reasonable so

Re: Help with huge index

2018-02-28 Thread Stuart Goldberg
I call deleteDocuments On Feb 28, 2018 8:16 PM, "Adrien Grand" wrote: > What do you mean by purging? What methods do you call? > > Le mer. 28 févr. 2018 à 19:34, Stuart Goldberg a > écrit : > > > I have huge lucene index. On disk it's about 24Gb. > > > > > > > > I have a purging routine that is

Re: Help with huge index

2018-02-28 Thread Adrien Grand
What do you mean by purging? What methods do you call? Le mer. 28 févr. 2018 à 19:34, Stuart Goldberg a écrit : > I have huge lucene index. On disk it's about 24Gb. > > > > I have a purging routine that is supposed to run and purge old docs. > > > > There are about 650 million docs in there and

Help with huge index

2018-02-28 Thread Stuart Goldberg
I have huge lucene index. On disk it's about 24Gb. I have a purging routine that is supposed to run and purge old docs. There are about 650 million docs in there and through testing I have determined that about 1/3 of these need to be purged. During the purge, every so often it's appare