The hang also only happens if you are using SerialMergeScheduler. Stu, one question: was there an interesting reason why you switched back to SerialMergeScheduler? Did you hit an issue with ConcurrentMergeScheduler?
Mike Stu Hood <[EMAIL PROTECTED]> wrote: > Hey gang, > > The finally block was not hiding any exceptions: I'm still seeing the hangs > in IndexWriter.close(). I'm appending the relevant output from > 'setInfoStream(System.out)'. It looks like after I order the close, the > IndexWriter begins a flush, and then hangs. > > Any ideas? > > Thanks, > Stu > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > IFD [main]: setInfoStream [EMAIL PROTECTED] > IW 0 [main]: setInfoStream: > dir=org.apache.lucene.store.FSDirectory@/data1/solr/solr_dovecot1208772000-1208808000_f0cad01b1cb2966a/index > autoCommit=true [EMAIL PROTECTED] [EMAIL PROTECTED] ramBufferSizeMB=16.0 > maxBuffereDocs=-1 maxBuffereDeleteTerms=-1 maxFieldLength=10000 > index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 > _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 > IW 0 [main]: setMergeScheduler [EMAIL PROTECTED] > >> Fetching > hdfs://hadoop.sat.mlsrvr.com:54310/user/hadoopuser/indexed/dovecot/1208807515-1208808145-1208808184/part-00004 > into a RAMDirectory: > >> .. done. Size: 22053888 > >> Merging directories to solr_dovecot1208772000-1208808000_f0cad01b1cb2966a > (factor: 4): > IW 0 [main]: flush at addIndexesNoOptimize > IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0 > flushDocs=false flushDeletes=false flushDocStores=false numDocs=0 > numBufDelTerms=0 > IW 0 [main]: index before flush _dn:C1349678 _j6:C1149270 _j5:C249194 > _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 > IW 0 [main]: now start transaction > IW 0 [main]: flush at startTransaction > IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0 > flushDocs=false flushDeletes=false flushDocStores=false numDocs=0 > numBufDelTerms=0 > IW 0 [main]: index before flush _dn:C1349678 _j6:C1149270 _j5:C249194 > _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 > IW 0 [main]: LMP: findMerges: 20 segments > IW 0 [main]: LMP: level 13.601146 to 14.351146: 2 segments > IW 0 [main]: LMP: level 12.390487 to 13.140487: 3 segments > IW 0 [main]: LMP: level 11.542906 to 12.292906: 3 segments > IW 0 [main]: LMP: level 10.656786 to 11.406786: 2 segments > IW 0 [main]: LMP: level -1.0 to 8.768208: 10 segments > IW 0 [main]: LMP: 10 to 14: add this merge > IW 0 [main]: LMP: 14 to 18: add this merge > IW 0 [main]: add merge to pendingMerges: _1f:Cx1000->_0 _1g:Cx1000->_0 > _1h:Cx1000->_0 _1i:Cx1000->_0 [total 1 pending] > IW 0 [main]: add merge to pendingMerges: _1j:Cx1000->_0 _1k:Cx1000->_0 > _1l:Cx1000->_0 _1m:Cx1000->_0 [total 2 pending] > IW 0 [main]: now merge > merge=_1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 _1i:Cx1000->_0 into _mq > index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 > _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 _1e:Cx50000->_0 _1f:Cx1000->_0 > _1g:Cx1000->_0 _1h:Cx1000->_0 _1i:Cx1000->_0 _1j:Cx1000->_0 _1k:Cx1000->_0 > _1l:Cx1000->_0 _1m:Cx1000->_0 _1n:Cx1000->_0 _1o:Cx857->_0 > IW 0 [main]: merging _1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 > _1i:Cx1000->_0 into _mq > IW 0 [main]: merge: total 4000 docs > IW 0 [main]: commitMerge _1f:Cx1000->_0 _1g:Cx1000->_0 _1h:Cx1000->_0 > _1i:Cx1000->_0 into _mq > IFD [main]: now checkpoint "segments_m" [17 segments ; isCommit = false] > IW 0 [main]: LMP: findMerges: 17 segments > IW 0 [main]: LMP: level 13.601146 to 14.351146: 2 segments > IW 0 [main]: LMP: level 12.390487 to 13.140487: 3 segments > IW 0 [main]: LMP: level 11.542906 to 12.292906: 3 segments > IW 0 [main]: LMP: level 10.656786 to 11.406786: 2 segments > IW 0 [main]: LMP: level -1.0 to 10.283832: 7 segments > IW 0 [main]: LMP: 10 to 14: add this merge > IW 0 [main]: now merge > merge=_1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 _1m:Cx1000->_0 into _mr > index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 > _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 _1e:Cx50000->_0 _mq:C4000 > _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 _1m:Cx1000->_0 _1n:Cx1000->_0 > _1o:Cx857->_0 > IW 0 [main]: merging _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 > _1m:Cx1000->_0 into _mr > IW 0 [main]: merge: total 4000 docs > IW 0 [main]: commitMerge _1j:Cx1000->_0 _1k:Cx1000->_0 _1l:Cx1000->_0 > _1m:Cx1000->_0 into _mr > IFD [main]: now checkpoint "segments_m" [14 segments ; isCommit = false] > IW 0 [main]: LMP: findMerges: 14 segments > IW 0 [main]: LMP: level 13.601146 to 14.351146: 2 segments > IW 0 [main]: LMP: level 12.390487 to 13.140487: 3 segments > IW 0 [main]: LMP: level 11.542906 to 12.292906: 3 segments > IW 0 [main]: LMP: level 10.656786 to 11.406786: 2 segments > IW 0 [main]: LMP: level -1.0 to 10.284627: 4 segments > IW 0 [main]: LMP: 10 to 14: add this merge > IW 0 [main]: add merge to pendingMerges: _mq:C4000 _mr:C4000 _1n:Cx1000->_0 > _1o:Cx857->_0 [total 1 pending] > IW 0 [main]: now merge > merge=_mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0 into _ms > index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 > _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 _1e:Cx50000->_0 _mq:C4000 _mr:C4000 > _1n:Cx1000->_0 _1o:Cx857->_0 > IW 0 [main]: merging _mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0 into > _ms > IW 0 [main]: merge: total 9857 docs > IW 0 [main]: commitMerge _mq:C4000 _mr:C4000 _1n:Cx1000->_0 _1o:Cx857->_0 > into _ms > IFD [main]: now checkpoint "segments_m" [11 segments ; isCommit = false] > IFD [main]: delete "_mq.fnm" > IFD [main]: delete "_mq.frq" > IFD [main]: delete "_mq.prx" > IFD [main]: delete "_mq.tis" > IFD [main]: delete "_mq.tii" > IFD [main]: delete "_mq.nrm" > IFD [main]: delete "_mq.fdx" > IFD [main]: delete "_mq.fdt" > IFD [main]: delete "_mr.fnm" > IFD [main]: delete "_mr.frq" > IFD [main]: delete "_mr.prx" > IFD [main]: delete "_mr.tis" > IFD [main]: delete "_mr.tii" > IFD [main]: delete "_mr.nrm" > IFD [main]: delete "_mr.fdx" > IFD [main]: delete "_mr.fdt" > IW 0 [main]: LMP: findMerges: 11 segments > IW 0 [main]: LMP: level 13.601146 to 14.351146: 2 segments > IW 0 [main]: LMP: level 12.390487 to 13.140487: 3 segments > IW 0 [main]: LMP: level 11.542906 to 12.292906: 3 segments > IW 0 [main]: LMP: level 10.656786 to 11.406786: 3 segments > IW 0 [main]: add merge to pendingMerges: _1e:Cx50000->_0 [total 1 pending] > IW 0 [main]: now merge > merge=_1e:Cx50000->_0 into _mt > index=_dn:C1349678 _j6:C1149270 _j5:C249194 _m4:C242243 _mf:C224430 > _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 _1e:Cx50000->_0 _ms:C9857 > IW 0 [main]: merging _1e:Cx50000->_0 into _mt > IW 0 [main]: merge: total 50000 docs > IW 0 [main]: commitMerge _1e:Cx50000->_0 into _mt > IFD [main]: now checkpoint "segments_m" [11 segments ; isCommit = false] > IW 0 [main]: LMP: findMerges: 11 segments > IW 0 [main]: LMP: level 13.601146 to 14.351146: 2 segments > IW 0 [main]: LMP: level 12.390487 to 13.140487: 3 segments > IW 0 [main]: LMP: level 11.542906 to 12.292906: 5 segments > IW 0 [main]: LMP: 5 to 9: add this merge > IW 0 [main]: LMP: level 10.339036 to 10.916957: 1 segments > IW 0 [main]: add merge to pendingMerges: _mk:C70616 _ml:C88 _mp:C50000 > _mo:C9905 [total 1 pending] > IW 0 [main]: now commit transaction > IW 0 [main]: checkpoint: wrote segments file "segments_n" > IFD [main]: now checkpoint "segments_n" [11 segments ; isCommit = true] > IFD [main]: deleteCommits: now remove commit "segments_m" > IFD [main]: delete "segments_m" > >> ... done. > >> Closing the target index... > IW 0 [main]: now flush at close > IW 0 [main]: flush: segment=null docStoreSegment=null docStoreOffset=0 > flushDocs=false flushDeletes=false flushDocStores=false numDocs=0 > numBufDelTerms=0 > IW 0 [main]: index before flush _dn:C1349678 _j6:C1149270 _j5:C249194 > _m4:C242243 _mf:C224430 _mk:C70616 _ml:C88 _mp:C50000 _mo:C9905 _mt:C50000 > _ms:C9857 > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > > -----Original Message----- > From: Stu Hood <[EMAIL PROTECTED]> > Sent: Sunday, April 20, 2008 3:41pm > To: java-user@lucene.apache.org > > > Subject: Re: Occasional Hang in IndexWriter.close() > > Hey Mike, > > Thanks for your input... the 'IndexWriter.close' call was actually in a > 'finally' block around the merge code, without a 'catch', which I realized > may have been hiding the exception (I didn't realize close would block if an > exception had occurred). > > I've moved the close out of the finally block, so hopefully if an exception > was being hidden, I'll see it and report back. > > Thanks, > Stu > > > > -----Original Message----- > From: Michael McCandless <[EMAIL PROTECTED]> > Sent: Saturday, April 19, 2008 4:40am > To: java-user@lucene.apache.org, [EMAIL PROTECTED] > Subject: Re: Occasional Hang in IndexWriter.close() > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
