Can you run CheckIndex (command line tool) and post the output?
How long does it take to open a reader on this same index, and perform
a simple query (eg TermQuery)?
Mike
On Wed, Nov 3, 2010 at 2:53 PM, Mark Kristensson
<mark.kristens...@smartsheet.com> wrote:
> I've successfully reproduced the issue in our lab with a copy from production
> and have broken the close() call into parts, as suggested, with one addition.
>
> Previously, the call was simply
>
> ...
> } finally {
> // Close
> if (indexWriter != null) {
> try {
> indexWriter.close();
> ...
>
>
> Now, that is broken into the various parts, including a prepareCommit();
>
> ...
> } finally {
> // Close
> if (indexWriter != null) {
> try {
> indexWriter.prepareCommit();
> Logger.debug("prepareCommit() complete");
> indexWriter.commit();
> Logger.debug("commit() complete");
> indexWriter.maybeMerge();
> Logger.debug("maybeMerge() complete");
> indexWriter.waitForMerges();
> Logger.debug("waitForMerges() complete");
> indexWriter.close();
> ...
>
>
> It turns out that the prepareCommit() is the slow call here, taking several
> seconds to complete.
>
> I've done some reading about it, but have not found anything that might be
> helpful here. The fact that it is slow every single time, even when I'm
> adding exactly one document to the index, is perplexing and leads to me to
> think something must be corrupt with the index.
>
> Furthermore, I tried optimizing the index to see if that would have any
> impact (I wasn't expecting much) and it did not.
>
> I'm stumped at this point and am thinking I may have to rebuild the index,
> though I would definitely prefer to avoid doing that and would like to know
> why this is happening.
>
> Thanks for your help,
> Mark
>
>
> On Nov 2, 2010, at 9:26 AM, Mark Kristensson wrote:
>
>>
>> Wonderful information on what happens during indexWriter.close(), thank you
>> very much! I've got some testing to do as a result.
>>
>> We are on Lucene 3.0.0 right now.
>>
>> One other detail that I neglected to mention is that the batch size does not
>> seem to have any relation to the time it takes to close on the index where
>> we are having issues. We've had batches add as few as 3 documents and
>> batches add as many as 2500 documents in the last hour and every single
>> close() call for that index takes 6 to 8 seconds. While I won't know until I
>> am able to individually test the different pieces of the close() operation,
>> I'd be very surprised if a batch that adds just 3 new documents results in
>> very much merge work being done. It seems as if there is some task happening
>> during merge that the indexWriter is never able to successfully complete and
>> so it tries to complete that task every single time close() is called.
>>
>> So, my working theory until I can dig deeper is that something is mildly
>> corrupt with the index (though not serious enough to affect most operations
>> on the index). Are there any good utilities for examining the health of an
>> index?
>>
>> I've dabbled with the experimental checkIndex object in the past (before we
>> upgraded to 3.0), but have found it to be incredibly slow and of marginal
>> value. Does anyone have any experience using CheckIndex to track down an
>> issue with a production index?
>>
>> Thanks again!
>> Mark
>>
>> On Nov 2, 2010, at 2:20 AM, Shai Erera wrote:
>>
>>> When you close IndexWriter, it performs several operations that might have a
>>> connection to the problem you describe:
>>>
>>> * Commit all the pending updates -- if your update batch size is more or
>>> less the same (i.e., comparable # of docs and total # bytes indexed), then
>>> you should not see a performance difference between closes.
>>>
>>> * Consults the MergePolicy and runs any merges it returns as candidates.
>>>
>>> * Waits for the merges to finish.
>>>
>>> Roughly, IndexWriter.close() can be substituted w/:
>>> writer.commit(false); // commits the changes, but does not run merges.
>>> writer.maybeMerge(); // runs merges returned by MergePolicy.
>>> writer.waitForMerges(); // if you use ConcurrentMergeScheduler, the above
>>> call returns immediately, not waiting for the merges to finish.
>>> writer.close(); // at this point, commit + merging has finished, so it does
>>> very little.
>>>
>>> As your index grows in size, so does its # of segments, and the segments
>>> size as well. So tweaking some parameters on the MergePolicy (such as
>>> mergeFactor, maxMergeMB etc.) can result in not attempting to merge too
>>> large segments.
>>>
>>> Alternatively, you can try the following:
>>>
>>> 1) Replace the call to writer.close() w/ the above sequence. Then, measure
>>> each call and report back which of them takes the suspicious amount of time.
>>>
>>> 2) Not running optimize() on a regular basis doesn't mean merges don't
>>> happen in the background. So if you want close() to return as fast as
>>> possible, you should call close(false). Note though that from time to time
>>> you should allow merges to finish, in order to reduce the # of segments.
>>>
>>> 3) If you want to control when the merges are run, you can open IndexWriter
>>> with NoMergePolicy, which always returns 0 merges to perform, or
>>> NoMergeScheduler which never executes merges. But be aware that this is
>>> dangerous as the # of segments in the index will continue to grow and search
>>> performance will degrade.
>>>
>>> The answers above is relevant for 3x, but most of them are also relevant for
>>> 2.9. If you have an older version of Lucene, then some of the solutions
>>> might still apply (such as close(false)).
>>>
>>> Hope this helps,
>>> Shai
>>>
>>> On Tue, Nov 2, 2010 at 12:55 AM, Mark Kristensson <
>>> mark.kristens...@smartsheet.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> One of our Lucene indexes has started misbehaving on indexWriter.close and
>>>> I'm searching for ideas about what may have happened and how to fix it.
>>>>
>>>> Here's our scenario:
>>>> - We have seven Lucene indexes that contain different sets of data from a
>>>> web application are indexed for searching by end users
>>>> - A java service runs twice a minute to pull changes from SQL DB queue
>>>> tables and update the relevant Lucene index(es)
>>>> - The two largest indexes (3.4GB and 3.8GB in size with 8 million and 6
>>>> million records, respectively) contain similar sets of data, but are
>>>> structured differently for different consumption (e.g. one has an All field
>>>> for general purpose searching, the other does not; one has numeric fields
>>>> for ranges, the other does not, etc.)
>>>> - We expunge deletes from our indexes twice per day
>>>> - A couple of weeks ago, one of the two large indexes became very slow to
>>>> close after each round of changes is applied by our indexing service.
>>>> Specifically, all of our indexes usually close in no more than 200
>>>> milliseconds, but this one index is now taking 6 to 8 seconds to close with
>>>> every single pass (and is leading to delays which are affecting end users).
>>>>
>>>> Questions from my attempts to troubleshoot the problem:
>>>> - Has anyone else seen behavior similar to this? What did you do to resolve
>>>> it?
>>>> - Does the size of an index or its documents (record count, disk size, avg
>>>> document size, max document size, etc.) have any correlation to the length
>>>> of time it takes to close an index?
>>>> - We are not currently optimizing any of our indexes on a regular basis,
>>>> could that have any impact upon indexing operations (e.g.
>>>> indexWriter.close())? My understanding is that optimization only affects
>>>> search performance, not indexing performance and to date we have not seen
>>>> any need to optimize based upon the performance of the search queries.
>>>>
>>>> Thanks,
>>>> Mark
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>>>
>>>>
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
