Hello all,
I have a very peculiar problem that is driving me crazy: on some of our
datasets and at some point in time during indexing, the merge operation
runs into a (semi-)infinite loop and keeps adding files to the index
until it runs out of free disk space.
The situation: I have an indexing application that uses Lucene 2.4.1.
Only one IndexWriter is involved, operating on a FSDirectory and using
the compound file format. The index is created from scratch. No
IndexReaders or IndexSearchers are open during indexing (double-checked
by adding explicit log statements where they are created).
For reasons unrelated to Lucene, the application is compiled with JET, a
commercial Java Windows compiler. A regular Java build has produced the
problem only once. The JET build does it every time - unless I keep
pressing F5 continuously in Windows Explorer on the index dir.
Here is what I see happening in the index dir:
- At first, it builds _0.cfs to _9.cfs without problems. The files vary
in size between 12 MB and 49 MB and add up to about 250 MB.
- Then, .del files are generated for some of these .cfs files. The
number of .del files and the .cfs files they correspond with differs
from time to time. I don't understand why these are created, as no
IndexReaders exist at this time.
- Then, it generates files called _a.fdt, _a.fdx, _a.fnm, _a.frq,
_a.nrm, _a.prx, _a.tii, _a.tis, _a.tvd, _a.tvf, _a.tvx. Together these
files add up to 219 MB. I assume this is the start of the merge of the
10 .cfs files and this is all still correct.
- Then, it generates _b files with those same extensions, then _c, _d,
etc. It only keeps generating new files, I never see files disappear.
The original .cfs files are still there.
- This continues until my hard drive is out of free space. At one test I
was at _8n and the index had grown from 250MB to 64 GB. Then the
application just hangs.
Interestingly, after killing the application in this test, there were
_8k.cfs and _8m.cfs files of 20 MB and 27 MB respectively. No other .cfs
files existed.
In some older threads on this list (e.g.
http://marc.info/?l=lucene-user&m=108300530413241&w=2) I read that
"Win32 seems to sometimes not permit one to delete a file immediately
after it has been closed". Could this explain the problem? Perhaps the
JET-compiled app gets to delete the file quicker than when the code is
running inside a Java VM and therefore runs into this issue? This would
also explain why pressing F5 during indexing lets the application
continue: external activity causing some manual delay.
At the end of this mail I have added the output of the InfoStream
installed on the IndexWriter, showing everything from the start to the
first few problematic merges. Lines starting with "===" are println's in
my own code to make sure that indeed only IndexWriters are generated and
no IndexSearchers/-Readers. The fun starts almost at the bottom, at the
first line containing "LMP: findMerges: 10 segments". The next lines
then get repeated over and over again, with different segment names. I
cannot explain why it mentions "1 deleted docIDs" a couple of lines
below the first "findMerges: 10" as no IndexWriter.deleteDocuments takes
place.
As you can see in this output, I am setting a SerialMergeScheduler to
rule out concurrency issues and making debugging easier. Both
SerialMergeScheduler and ConcurrentMergeScheduler give this problem though.
I would be grateful if anyone could pose some light on this or can
advise me on what I can try next. I even considered hacking the
FSDirectory code and adding some delay in the deleteFile operation to
see if the above-mentioned win32 issue is the problem, but unless you
know what you're doing, such hacks can even cause such problems in the
first place.
Kind regards,
Chris
--
=== Creating new FSDirectory
=== Opening indexWriter (create=true)
IFD [AWT-EventQueue-0]: setInfoStream
deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@9fb0688
IW 0 [AWT-EventQueue-0]: setInfoStream:
dir=org.apache.lucene.store.fsdirect...@d:\index autoCommit=false
mergepolicy=org.apache.lucene.index.logbytesizemergepol...@a11d0d8
mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@a3fa9d8
ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1
maxFieldLength=2147483647 index=
IW 0 [AWT-EventQueue-0]: setMergeScheduler
org.apache.lucene.index.serialmergeschedu...@a3d1fe8
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=46.015 vs
trigger=16 allocMB=46.015 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=46.015
allocMB=46.015
IW 0 [CrawlThread]: flush: segment=_0 docStoreSegment=_0
docStoreOffset=0 flushDocs=true flushDeletes=false flushDocStores=false
numDocs=2193 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush
IW 0 [CrawlThread]: DW: flush postings as segment _0 numDocs=2193
IW 0 [CrawlThread]: DW: oldRAMSize=48249856 newFlushedSize=13379990
docs/MB=171.863 new/old=27.731%
IFD [CrawlThread]: now checkpoint "segments_1" [1 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [1 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_0.fnm"
IFD [CrawlThread]: delete "_0.frq"
IFD [CrawlThread]: delete "_0.prx"
IFD [CrawlThread]: delete "_0.tis"
IFD [CrawlThread]: delete "_0.tii"
IFD [CrawlThread]: delete "_0.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 1 segments
IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 1 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs
trigger=16 allocMB=46.015 vs trigger=16.8 byteBlockFree=12.438
charBlockFree=8.969
IW 0 [CrawlThread]: DW: after free: freedMB=30.841 usedMB=0.044
allocMB=15.173
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=6.806 vs
trigger=16 allocMB=16.829 vs trigger=16.8 byteBlockFree=0.812
charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=6.806
allocMB=15.173
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.242 vs
trigger=16 allocMB=16.861 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.687 usedMB=9.242
allocMB=15.174
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=43.339 vs
trigger=16 allocMB=43.401 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=43.339
allocMB=43.401
IW 0 [CrawlThread]: flush: segment=_1 docStoreSegment=_0
docStoreOffset=2193 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=1799 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0
IW 0 [CrawlThread]: DW: flush postings as segment _1 numDocs=1799
IW 0 [CrawlThread]: DW: oldRAMSize=45444096 newFlushedSize=12691373
docs/MB=148.635 new/old=27.927%
IFD [CrawlThread]: now checkpoint "segments_1" [2 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [2 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_1.fnm"
IFD [CrawlThread]: delete "_1.frq"
IFD [CrawlThread]: delete "_1.prx"
IFD [CrawlThread]: delete "_1.tis"
IFD [CrawlThread]: delete "_1.tii"
IFD [CrawlThread]: delete "_1.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 2 segments
IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 2 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs
trigger=16 allocMB=43.401 vs trigger=16.8 byteBlockFree=11.781
charBlockFree=8.469
IW 0 [CrawlThread]: DW: after free: freedMB=28.217 usedMB=0.044
allocMB=15.185
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=40.889 vs
trigger=16 allocMB=40.951 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=40.889
allocMB=40.951
IW 0 [CrawlThread]: flush: segment=_2 docStoreSegment=_0
docStoreOffset=3992 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=1013 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
IW 0 [CrawlThread]: DW: flush postings as segment _2 numDocs=1013
IW 0 [CrawlThread]: DW: oldRAMSize=42874880 newFlushedSize=11684845
docs/MB=90.905 new/old=27.253%
IFD [CrawlThread]: now checkpoint "segments_1" [3 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [3 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_2.fnm"
IFD [CrawlThread]: delete "_2.frq"
IFD [CrawlThread]: delete "_2.prx"
IFD [CrawlThread]: delete "_2.tis"
IFD [CrawlThread]: delete "_2.tii"
IFD [CrawlThread]: delete "_2.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 3 segments
IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 3 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs
trigger=16 allocMB=40.951 vs trigger=16.8 byteBlockFree=10.688
charBlockFree=8.062
IW 0 [CrawlThread]: DW: after free: freedMB=25.779 usedMB=0.044
allocMB=15.172
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=38.82 vs
trigger=16 allocMB=38.883 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=38.82
allocMB=38.883
IW 0 [CrawlThread]: flush: segment=_3 docStoreSegment=_0
docStoreOffset=5005 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=427 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0
IW 0 [CrawlThread]: DW: flush postings as segment _3 numDocs=427
IW 0 [CrawlThread]: DW: oldRAMSize=40706048 newFlushedSize=10574425
docs/MB=42.342 new/old=25.978%
IFD [CrawlThread]: now checkpoint "segments_1" [4 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [4 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_3.fnm"
IFD [CrawlThread]: delete "_3.frq"
IFD [CrawlThread]: delete "_3.prx"
IFD [CrawlThread]: delete "_3.tis"
IFD [CrawlThread]: delete "_3.tii"
IFD [CrawlThread]: delete "_3.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 4 segments
IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 4 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs
trigger=16 allocMB=38.883 vs trigger=16.8 byteBlockFree=9.312
charBlockFree=7.812
IW 0 [CrawlThread]: DW: after free: freedMB=23.686 usedMB=0.044
allocMB=15.197
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=8.588 vs
trigger=16 allocMB=16.822 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=8.588
allocMB=15.197
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=16.15 vs
trigger=16 allocMB=19.51 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=3.297 usedMB=16.15
allocMB=16.213
IW 0 [CrawlThread]: flush: segment=_4 docStoreSegment=_0
docStoreOffset=5432 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=1145 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0
IW 0 [CrawlThread]: DW: flush postings as segment _4 numDocs=1145
IW 0 [CrawlThread]: DW: oldRAMSize=16934912 newFlushedSize=5040651
docs/MB=238.187 new/old=29.765%
IFD [CrawlThread]: now checkpoint "segments_1" [5 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [5 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_4.fnm"
IFD [CrawlThread]: delete "_4.frq"
IFD [CrawlThread]: delete "_4.prx"
IFD [CrawlThread]: delete "_4.tis"
IFD [CrawlThread]: delete "_4.tii"
IFD [CrawlThread]: delete "_4.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 5 segments
IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 5 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=45.533 vs
trigger=16 allocMB=45.596 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=45.533
allocMB=45.596
IW 0 [CrawlThread]: flush: segment=_5 docStoreSegment=_0
docStoreOffset=6577 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=808 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0 _4:c1145->_0
IW 0 [CrawlThread]: DW: flush postings as segment _5 numDocs=808
IW 0 [CrawlThread]: DW: oldRAMSize=47745024 newFlushedSize=12761396
docs/MB=66.392 new/old=26.728%
IFD [CrawlThread]: now checkpoint "segments_1" [6 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [6 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_5.fnm"
IFD [CrawlThread]: delete "_5.frq"
IFD [CrawlThread]: delete "_5.prx"
IFD [CrawlThread]: delete "_5.tis"
IFD [CrawlThread]: delete "_5.tii"
IFD [CrawlThread]: delete "_5.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 6 segments
IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 6 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs
trigger=16 allocMB=45.596 vs trigger=16.8 byteBlockFree=11.5
charBlockFree=9.031
IW 0 [CrawlThread]: DW: after free: freedMB=30.404 usedMB=0.044
allocMB=15.192
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=7.051 vs
trigger=16 allocMB=16.848 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=7.051
allocMB=15.192
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.829 vs
trigger=16 allocMB=16.942 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.75 usedMB=9.829
allocMB=15.192
IW 0 [CrawlThread]: hit exception adding document
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=48.622 vs
trigger=16 allocMB=48.685 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull
IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=48.622
allocMB=48.685
IW 0 [CrawlThread]: flush: segment=_6 docStoreSegment=_0
docStoreOffset=7385 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=1689 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0
IW 0 [CrawlThread]: DW: flush postings as segment _6 numDocs=1689
IW 0 [CrawlThread]: DW: oldRAMSize=50983936 newFlushedSize=14095657
docs/MB=125.645 new/old=27.647%
IFD [CrawlThread]: now checkpoint "segments_1" [7 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [7 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_6.fnm"
IFD [CrawlThread]: delete "_6.frq"
IFD [CrawlThread]: delete "_6.prx"
IFD [CrawlThread]: delete "_6.tis"
IFD [CrawlThread]: delete "_6.tii"
IFD [CrawlThread]: delete "_6.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 7 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 7 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs
trigger=16 allocMB=48.685 vs trigger=16.8 byteBlockFree=12.969
charBlockFree=9.594
IW 0 [CrawlThread]: DW: after free: freedMB=33.498 usedMB=0.044
allocMB=15.187
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=7.007 vs
trigger=16 allocMB=16.812 vs trigger=16.8 byteBlockFree=0.281
charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=7.007
allocMB=15.187
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.254 vs
trigger=16 allocMB=16.812 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=9.254
allocMB=15.188
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=11.913 vs
trigger=16 allocMB=16.844 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=11.913
allocMB=15.188
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=14.566 vs
trigger=16 allocMB=16.813 vs trigger=16.8 byteBlockFree=0 charBlockFree=0
IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=14.566
allocMB=15.189
IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.151 allocMB=16.214
triggerMB=16
IW 0 [CrawlThread]: flush: segment=_7 docStoreSegment=_0
docStoreOffset=9074 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=2941 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0
IW 0 [CrawlThread]: DW: flush postings as segment _7 numDocs=2941
IW 0 [CrawlThread]: DW: oldRAMSize=16935936 newFlushedSize=5920901
docs/MB=520.843 new/old=34.961%
IFD [CrawlThread]: now checkpoint "segments_1" [8 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [8 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_7.fnm"
IFD [CrawlThread]: delete "_7.frq"
IFD [CrawlThread]: delete "_7.prx"
IFD [CrawlThread]: delete "_7.tis"
IFD [CrawlThread]: delete "_7.tii"
IFD [CrawlThread]: delete "_7.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 8 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 8 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=14.513 vs
trigger=16 allocMB=16.808 vs trigger=16.8 byteBlockFree=0 charBlockFree=0.75
IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=14.513
allocMB=15.183
IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.057 allocMB=16.277
triggerMB=16
IW 0 [CrawlThread]: flush: segment=_8 docStoreSegment=_0
docStoreOffset=12015 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=3171 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0
IW 0 [CrawlThread]: DW: flush postings as segment _8 numDocs=3171
IW 0 [CrawlThread]: DW: oldRAMSize=16836608 newFlushedSize=7011110
docs/MB=474.252 new/old=41.642%
IFD [CrawlThread]: now checkpoint "segments_1" [9 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [9 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_8.fnm"
IFD [CrawlThread]: delete "_8.frq"
IFD [CrawlThread]: delete "_8.prx"
IFD [CrawlThread]: delete "_8.tis"
IFD [CrawlThread]: delete "_8.tii"
IFD [CrawlThread]: delete "_8.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 9 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 9 segments
IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=15.678 vs
trigger=16 allocMB=17.178 vs trigger=16.8 byteBlockFree=1.438
charBlockFree=0
IW 0 [CrawlThread]: DW: nothing to free
IW 0 [CrawlThread]: DW: after free: freedMB=1.438 usedMB=15.678
allocMB=15.74
IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.067 allocMB=16.13
triggerMB=16
IW 0 [CrawlThread]: flush: segment=_9 docStoreSegment=_0
docStoreOffset=15186 flushDocs=true flushDeletes=false
flushDocStores=false numDocs=3401 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0
_7:c2941->_0 _8:c3171->_0
IW 0 [CrawlThread]: DW: flush postings as segment _9 numDocs=3401
IW 0 [CrawlThread]: DW: oldRAMSize=16847872 newFlushedSize=6184635
docs/MB=576.624 new/old=36.709%
IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit =
false]
IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_9.fnm"
IFD [CrawlThread]: delete "_9.frq"
IFD [CrawlThread]: delete "_9.prx"
IFD [CrawlThread]: delete "_9.tis"
IFD [CrawlThread]: delete "_9.tii"
IFD [CrawlThread]: delete "_9.nrm"
IW 0 [CrawlThread]: LMP: findMerges: 10 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments
IW 0 [CrawlThread]: LMP: 0 to 10: add this merge
IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0
_1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0
_6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending]
IW 0 [CrawlThread]: DW: apply 0 buffered deleted terms and 1 deleted
docIDs and 0 deleted queries on 10 segments.
IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit =
false]
IW 0 [CrawlThread]: now flush at merge
IW 0 [CrawlThread]: flush: segment=null docStoreSegment=_0
docStoreOffset=18587 flushDocs=false flushDeletes=false
flushDocStores=true numDocs=0 numBufDelTerms=0
IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0
_2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0
_7:c2941->_0 _8:c3171->_0 _9:c3401->_0
IW 0 [CrawlThread]: flush shared docStore segment _0
IW 0 [CrawlThread]: DW: closeDocStore: 2 files to flush to segment _0
numDocs=18587
IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit =
false]
IFD [CrawlThread]: delete "_0.fdt"
IFD [CrawlThread]: delete "_0.fdx"
IFD [CrawlThread]: delete "_0.fdt"
IFD [CrawlThread]: delete "_0.fdx"
IW 0 [CrawlThread]: now merge
merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _a
merge=org.apache.lucene.index.mergepolicy$oneme...@b590b98
index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0
IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0
_3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0
_8:c3171->_0 _9:c3401->_0 into _a
IW 0 [CrawlThread]: merge: total 18586 docs
IW 0 [CrawlThread]: LMP: findMerges: 10 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments
IW 0 [CrawlThread]: LMP: 0 to 10: add this merge
IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0
_1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0
_6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending]
IW 0 [CrawlThread]: now merge
merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _b
merge=org.apache.lucene.index.mergepolicy$oneme...@b590658
index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0
IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0
_3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0
_8:c3171->_0 _9:c3401->_0 into _b
IW 0 [CrawlThread]: merge: total 18586 docs
IW 0 [CrawlThread]: LMP: findMerges: 10 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments
IW 0 [CrawlThread]: LMP: 0 to 10: add this merge
IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0
_1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0
_6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending]
IW 0 [CrawlThread]: now merge
merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _c
merge=org.apache.lucene.index.mergepolicy$oneme...@b5906b8
index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0
IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0
_3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0
_8:c3171->_0 _9:c3401->_0 into _c
IW 0 [CrawlThread]: merge: total 18586 docs
IW 0 [CrawlThread]: LMP: findMerges: 10 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments
IW 0 [CrawlThread]: LMP: 0 to 10: add this merge
IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0
_1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0
_6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending]
IW 0 [CrawlThread]: now merge
merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _d
merge=org.apache.lucene.index.mergepolicy$oneme...@b5906e8
index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0
IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0
_3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0
_8:c3171->_0 _9:c3401->_0 into _d
IW 0 [CrawlThread]: merge: total 18586 docs
IW 0 [CrawlThread]: LMP: findMerges: 10 segments
IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments
IW 0 [CrawlThread]: LMP: 0 to 10: add this merge
IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0
_1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0
_6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 [total 1 pending]
IW 0 [CrawlThread]: now merge
merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _e
merge=org.apache.lucene.index.mergepolicy$oneme...@b590688
index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0
_5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0
.......
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
