[
https://issues.apache.org/jira/browse/LUCENE-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655545#comment-13655545
]
l0co commented on LUCENE-3422:
------------------------------
I have the similar problems in 3.1.0, likewise here and LUCENE-1638. I believe
there's a bug in lucene synchronization/multi threading execution model during
merge.
I use the Hibernate Search with exclusive index usage enabled, what means that
the same IndexWriter is used constantly per index dir and only closed for some
reasons. On the other hand the IndexWriter uses ConcurrentMergeScheduler by
default. I occasionally have FileNotFoundException telling me that some segment
files are not found (FNFE).
When I look into the implementation I can see in merge:
{code}
void merge() {
mergeInit(merge);
mergeMiddle(merge);
mergeSuccess(merge);
// is the problem here (?)
mergeFinish(merge);
}
{code}
The merge() method is not synchronized at all. I believe that in first three
lines (before "is the problem here?" line) the merge is started and scheduled,
and can do some job (like remove the segment files). However, the segmentInfo
update is only done in mergeFinish() method (where it waits for all merges and
updates segmentInfo-s then). If in the "is the problem here?" line other thread
invokes doFlush() (which is synchronized) we have the IndexWriter in the
partial or completely done merge (some files are removed), but it has still
outdated segmentsInfo, which will be updated for a while in mergeFinish().
This might be a wrong interpretation because I didn't thoroughly review the
code, but the error is there. It might occasionally produce FNFE on doFlush()
because of outated segmentInfos regarding the current directory state in
filesystem (this has been verified).
> IndeIndexWriter.optimize() throws FileNotFoundException and IOException
> -----------------------------------------------------------------------
>
> Key: LUCENE-3422
> URL: https://issues.apache.org/jira/browse/LUCENE-3422
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Elizabeth Nisha
>
> I am using lucene 3.0.2 search APIs for my application.
> Indexed data is about 350MB and time taken for indexing is 25 hrs. Search
> indexing and Optimization runs in two different threads. Optimization runs
> for every 1 hour and it doesn't run while indexing is going on and vice
> versa. When optimization is going on using IndexWriter.optimize(),
> FileNotFoundException and IOException are seen in my log and the index file
> is getting corrupted, log says
> 1. java.io.IOException: No sub-file with id _5r8.fdt found
> [The file name in this message changes over time (_5r8.fdt, _6fa.fdt,
> _6uh.fdt, ..., _emv.fdt) ]
> 2. java.io.FileNotFoundException:
> /local/groups/necim/index_5.3/index/_bdx.cfs (No such file or directory)
> 3. java.io.FileNotFoundException:
> /local/groups/necim/index_5.3/index/_hkq.cfs (No such file or directory)
> Stack trace: java.io.IOException: background merge hit exception:
> _hkp:c100->_hkp _hkq:c100->_hkp _hkr:c100->_hkr _hks:c100->_hkr _hxb:c5500
> _hx5:c1000 _hxc:c198
> 84 into _hxd [optimize] [mergeDocStores]
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2359)
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2298)
> at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2268)
> at com.telelogic.cs.search.SearchIndex.doOptimize(SearchIndex.java:130)
> at
> com.telelogic.cs.search.SearchIndexerThread$1.run(SearchIndexerThread.java:337)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.FileNotFoundException:
> /local/groups/necim/index_5.3/index/_hkq.cfs (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:76)
> at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:97)
> at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:87)
> at
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)
> at
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:67)
> at
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:114)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4309)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3965)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:231)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:288)
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
