Shalin Shekhar Mangar wrote:

Let me try to explain.

I have a master where indexing is done. I have multiple slaves for querying.

If I commit+optimize on the master and then rsync the index, the data
transferred on the network is huge. An alternate way is to commit on master, transfer the delta to the slave and issue an optimize on the slave. This is
very fast because less data is transferred on the network.

Large segment merges will also send huge traffic. You may just want to send all updates (document adds/deletes) to all slaves directly? It'd be nice if you could somehow NOT sync the effects of segment merging, but do sync doc add/deletes... not sure how to do that.

However, we need to ensure that the index on the slave is actually in sync with the master. So that on another commit, we can blindly transfer the
delta to the slave.

I assume your app ensures that no deltas arrive to the slave while it's running optimize? So then the question becomes (I think) "if two indices are identical to begin with, and I separately run optimize on each, will the resulting two optimized indices be identical?".

By "in sync" you also require the final segment name (after optimize) is identical right?

I think the answer is yes, but I'm not certain unless I think more about it. Also this behavior is not "promised" in Lucene's API.

Merges for optimize are now allowed to run concurrently (by default, with ConcurrentMergeScheduler), except for the final (< mergeFactor segments) merge, which waits until others have finished. So if there are 7 obvious merges necessary to optimize, 3 will run concurrently, while 4 wait. Those 4 then run as the merges finish over time, which may happen in different orders for each index and so different merges may run. Then the final merge will run and I *think* the net number of merges that ran should always be the same and so the final segment name should be the same.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Reply via email to