> On Unix, one can "kill -SIGQUIT" to get a thread stack trace dump for > all threads; do you know how to do this on Windows? If so, can you do > that at the end when IW starts doing this infinite merging? That > would be very helpful towards understanding why this recursion is > happening (though it is spooky that this is all happening under > JET...)
On Windows, press Ctrl-Break in the console, this sends SIGBREAK to JVM, which does the same on Windows (the stack traces will appear in console). Ctrl-Break is *not* Ctrl-C (which kills the process). Break is the seldom used key also known as "Pause" key somewhere right top on the keyboard. Uwe > On Tue, Apr 14, 2009 at 5:28 AM, Christiaan Fluit > <christiaan.fl...@aduna-software.com> wrote: > > Hello all, > > > > I have a very peculiar problem that is driving me crazy: on some of our > > datasets and at some point in time during indexing, the merge operation > runs > > into a (semi-)infinite loop and keeps adding files to the index until it > > runs out of free disk space. > > > > The situation: I have an indexing application that uses Lucene 2.4.1. > Only > > one IndexWriter is involved, operating on a FSDirectory and using the > > compound file format. The index is created from scratch. No IndexReaders > or > > IndexSearchers are open during indexing (double-checked by adding > explicit > > log statements where they are created). > > > > For reasons unrelated to Lucene, the application is compiled with JET, a > > commercial Java Windows compiler. A regular Java build has produced the > > problem only once. The JET build does it every time - unless I keep > pressing > > F5 continuously in Windows Explorer on the index dir. > > > > Here is what I see happening in the index dir: > > > > - At first, it builds _0.cfs to _9.cfs without problems. The files vary > in > > size between 12 MB and 49 MB and add up to about 250 MB. > > > > - Then, .del files are generated for some of these .cfs files. The > number of > > .del files and the .cfs files they correspond with differs from time to > > time. I don't understand why these are created, as no IndexReaders exist > at > > this time. > > > > - Then, it generates files called _a.fdt, _a.fdx, _a.fnm, _a.frq, > _a.nrm, > > _a.prx, _a.tii, _a.tis, _a.tvd, _a.tvf, _a.tvx. Together these files add > up > > to 219 MB. I assume this is the start of the merge of the 10 .cfs files > and > > this is all still correct. > > > > - Then, it generates _b files with those same extensions, then _c, _d, > etc. > > It only keeps generating new files, I never see files disappear. The > > original .cfs files are still there. > > > > - This continues until my hard drive is out of free space. At one test I > was > > at _8n and the index had grown from 250MB to 64 GB. Then the application > > just hangs. > > > > Interestingly, after killing the application in this test, there were > > _8k.cfs and _8m.cfs files of 20 MB and 27 MB respectively. No other .cfs > > files existed. > > > > In some older threads on this list (e.g. > > http://marc.info/?l=lucene-user&m=108300530413241&w=2) I read that > "Win32 > > seems to sometimes not permit one to delete a file immediately after it > has > > been closed". Could this explain the problem? Perhaps the JET-compiled > app > > gets to delete the file quicker than when the code is running inside a > Java > > VM and therefore runs into this issue? This would also explain why > pressing > > F5 during indexing lets the application continue: external activity > causing > > some manual delay. > > > > At the end of this mail I have added the output of the InfoStream > installed > > on the IndexWriter, showing everything from the start to the first few > > problematic merges. Lines starting with "===" are println's in my own > code > > to make sure that indeed only IndexWriters are generated and no > > IndexSearchers/-Readers. The fun starts almost at the bottom, at the > first > > line containing "LMP: findMerges: 10 segments". The next lines then get > > repeated over and over again, with different segment names. I cannot > explain > > why it mentions "1 deleted docIDs" a couple of lines below the first > > "findMerges: 10" as no IndexWriter.deleteDocuments takes place. > > > > As you can see in this output, I am setting a SerialMergeScheduler to > rule > > out concurrency issues and making debugging easier. Both > > SerialMergeScheduler and ConcurrentMergeScheduler give this problem > though. > > > > I would be grateful if anyone could pose some light on this or can > advise me > > on what I can try next. I even considered hacking the FSDirectory code > and > > adding some delay in the deleteFile operation to see if the above- > mentioned > > win32 issue is the problem, but unless you know what you're doing, such > > hacks can even cause such problems in the first place. > > > > > > Kind regards, > > > > Chris > > -- > > > > === Creating new FSDirectory > > === Opening indexWriter (create=true) > > IFD [AWT-EventQueue-0]: setInfoStream > > > deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@9f > b0688 > > IW 0 [AWT-EventQueue-0]: setInfoStream: > > dir=org.apache.lucene.store.fsdirect...@d:\index autoCommit=false > > mergepolicy=org.apache.lucene.index.logbytesizemergepol...@a11d0d8 > > mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@a3fa9d8 > > ramBufferSizeMB=16.0 maxBufferedDocs=-1 maxBuffereDeleteTerms=-1 > > maxFieldLength=2147483647 index= > > IW 0 [AWT-EventQueue-0]: setMergeScheduler > > org.apache.lucene.index.serialmergeschedu...@a3d1fe8 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=46.015 vs > > trigger=16 allocMB=46.015 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=46.015 > > allocMB=46.015 > > IW 0 [CrawlThread]: flush: segment=_0 docStoreSegment=_0 > docStoreOffset=0 > > flushDocs=true flushDeletes=false flushDocStores=false numDocs=2193 > > numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush > > IW 0 [CrawlThread]: DW: flush postings as segment _0 numDocs=2193 > > IW 0 [CrawlThread]: DW: oldRAMSize=48249856 newFlushedSize=13379990 > > docs/MB=171.863 new/old=27.731% > > IFD [CrawlThread]: now checkpoint "segments_1" [1 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [1 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_0.fnm" > > IFD [CrawlThread]: delete "_0.frq" > > IFD [CrawlThread]: delete "_0.prx" > > IFD [CrawlThread]: delete "_0.tis" > > IFD [CrawlThread]: delete "_0.tii" > > IFD [CrawlThread]: delete "_0.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 1 segments > > IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 1 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs > > trigger=16 allocMB=46.015 vs trigger=16.8 byteBlockFree=12.438 > > charBlockFree=8.969 > > IW 0 [CrawlThread]: DW: after free: freedMB=30.841 usedMB=0.044 > > allocMB=15.173 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=6.806 vs > > trigger=16 allocMB=16.829 vs trigger=16.8 byteBlockFree=0.812 > > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=6.806 > > allocMB=15.173 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.242 vs > > trigger=16 allocMB=16.861 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.687 usedMB=9.242 > > allocMB=15.174 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=43.339 vs > > trigger=16 allocMB=43.401 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=43.339 > > allocMB=43.401 > > IW 0 [CrawlThread]: flush: segment=_1 docStoreSegment=_0 > > docStoreOffset=2193 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=1799 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _1 numDocs=1799 > > IW 0 [CrawlThread]: DW: oldRAMSize=45444096 newFlushedSize=12691373 > > docs/MB=148.635 new/old=27.927% > > IFD [CrawlThread]: now checkpoint "segments_1" [2 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [2 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_1.fnm" > > IFD [CrawlThread]: delete "_1.frq" > > IFD [CrawlThread]: delete "_1.prx" > > IFD [CrawlThread]: delete "_1.tis" > > IFD [CrawlThread]: delete "_1.tii" > > IFD [CrawlThread]: delete "_1.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 2 segments > > IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 2 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs > > trigger=16 allocMB=43.401 vs trigger=16.8 byteBlockFree=11.781 > > charBlockFree=8.469 > > IW 0 [CrawlThread]: DW: after free: freedMB=28.217 usedMB=0.044 > > allocMB=15.185 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=40.889 vs > > trigger=16 allocMB=40.951 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=40.889 > > allocMB=40.951 > > IW 0 [CrawlThread]: flush: segment=_2 docStoreSegment=_0 > > docStoreOffset=3992 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=1013 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _2 numDocs=1013 > > IW 0 [CrawlThread]: DW: oldRAMSize=42874880 newFlushedSize=11684845 > > docs/MB=90.905 new/old=27.253% > > IFD [CrawlThread]: now checkpoint "segments_1" [3 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [3 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_2.fnm" > > IFD [CrawlThread]: delete "_2.frq" > > IFD [CrawlThread]: delete "_2.prx" > > IFD [CrawlThread]: delete "_2.tis" > > IFD [CrawlThread]: delete "_2.tii" > > IFD [CrawlThread]: delete "_2.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 3 segments > > IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 3 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs > > trigger=16 allocMB=40.951 vs trigger=16.8 byteBlockFree=10.688 > > charBlockFree=8.062 > > IW 0 [CrawlThread]: DW: after free: freedMB=25.779 usedMB=0.044 > > allocMB=15.172 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=38.82 vs > > trigger=16 allocMB=38.883 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=38.82 > > allocMB=38.883 > > IW 0 [CrawlThread]: flush: segment=_3 docStoreSegment=_0 > > docStoreOffset=5005 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=427 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _3 numDocs=427 > > IW 0 [CrawlThread]: DW: oldRAMSize=40706048 newFlushedSize=10574425 > > docs/MB=42.342 new/old=25.978% > > IFD [CrawlThread]: now checkpoint "segments_1" [4 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [4 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_3.fnm" > > IFD [CrawlThread]: delete "_3.frq" > > IFD [CrawlThread]: delete "_3.prx" > > IFD [CrawlThread]: delete "_3.tis" > > IFD [CrawlThread]: delete "_3.tii" > > IFD [CrawlThread]: delete "_3.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 4 segments > > IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 4 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs > > trigger=16 allocMB=38.883 vs trigger=16.8 byteBlockFree=9.312 > > charBlockFree=7.812 > > IW 0 [CrawlThread]: DW: after free: freedMB=23.686 usedMB=0.044 > > allocMB=15.197 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=8.588 vs > > trigger=16 allocMB=16.822 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=8.588 > > allocMB=15.197 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=16.15 vs > > trigger=16 allocMB=19.51 vs trigger=16.8 byteBlockFree=0 charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=3.297 usedMB=16.15 > > allocMB=16.213 > > IW 0 [CrawlThread]: flush: segment=_4 docStoreSegment=_0 > > docStoreOffset=5432 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=1145 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _4 numDocs=1145 > > IW 0 [CrawlThread]: DW: oldRAMSize=16934912 newFlushedSize=5040651 > > docs/MB=238.187 new/old=29.765% > > IFD [CrawlThread]: now checkpoint "segments_1" [5 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [5 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_4.fnm" > > IFD [CrawlThread]: delete "_4.frq" > > IFD [CrawlThread]: delete "_4.prx" > > IFD [CrawlThread]: delete "_4.tis" > > IFD [CrawlThread]: delete "_4.tii" > > IFD [CrawlThread]: delete "_4.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 5 segments > > IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 5 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=45.533 vs > > trigger=16 allocMB=45.596 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=45.533 > > allocMB=45.596 > > IW 0 [CrawlThread]: flush: segment=_5 docStoreSegment=_0 > > docStoreOffset=6577 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=808 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _5 numDocs=808 > > IW 0 [CrawlThread]: DW: oldRAMSize=47745024 newFlushedSize=12761396 > > docs/MB=66.392 new/old=26.728% > > IFD [CrawlThread]: now checkpoint "segments_1" [6 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [6 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_5.fnm" > > IFD [CrawlThread]: delete "_5.frq" > > IFD [CrawlThread]: delete "_5.prx" > > IFD [CrawlThread]: delete "_5.tis" > > IFD [CrawlThread]: delete "_5.tii" > > IFD [CrawlThread]: delete "_5.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 6 segments > > IW 0 [CrawlThread]: LMP: level 6.37646 to 7.12646: 6 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs > > trigger=16 allocMB=45.596 vs trigger=16.8 byteBlockFree=11.5 > > charBlockFree=9.031 > > IW 0 [CrawlThread]: DW: after free: freedMB=30.404 usedMB=0.044 > > allocMB=15.192 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=7.051 vs > > trigger=16 allocMB=16.848 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=7.051 > > allocMB=15.192 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.829 vs > > trigger=16 allocMB=16.942 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.75 usedMB=9.829 > > allocMB=15.192 > > IW 0 [CrawlThread]: hit exception adding document > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=48.622 vs > > trigger=16 allocMB=48.685 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free; now set bufferIsFull > > IW 0 [CrawlThread]: DW: after free: freedMB=0 usedMB=48.622 > > allocMB=48.685 > > IW 0 [CrawlThread]: flush: segment=_6 docStoreSegment=_0 > > docStoreOffset=7385 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=1689 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _6 numDocs=1689 > > IW 0 [CrawlThread]: DW: oldRAMSize=50983936 newFlushedSize=14095657 > > docs/MB=125.645 new/old=27.647% > > IFD [CrawlThread]: now checkpoint "segments_1" [7 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [7 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_6.fnm" > > IFD [CrawlThread]: delete "_6.frq" > > IFD [CrawlThread]: delete "_6.prx" > > IFD [CrawlThread]: delete "_6.tis" > > IFD [CrawlThread]: delete "_6.tii" > > IFD [CrawlThread]: delete "_6.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 7 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 7 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=0.044 vs > > trigger=16 allocMB=48.685 vs trigger=16.8 byteBlockFree=12.969 > > charBlockFree=9.594 > > IW 0 [CrawlThread]: DW: after free: freedMB=33.498 usedMB=0.044 > > allocMB=15.187 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=7.007 vs > > trigger=16 allocMB=16.812 vs trigger=16.8 byteBlockFree=0.281 > > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=7.007 > > allocMB=15.187 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=9.254 vs > > trigger=16 allocMB=16.812 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=9.254 > > allocMB=15.188 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=11.913 vs > > trigger=16 allocMB=16.844 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.656 usedMB=11.913 > > allocMB=15.188 > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=14.566 vs > > trigger=16 allocMB=16.813 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=14.566 > > allocMB=15.189 > > IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.151 allocMB=16.214 > > triggerMB=16 > > IW 0 [CrawlThread]: flush: segment=_7 docStoreSegment=_0 > > docStoreOffset=9074 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=2941 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _7 numDocs=2941 > > IW 0 [CrawlThread]: DW: oldRAMSize=16935936 newFlushedSize=5920901 > > docs/MB=520.843 new/old=34.961% > > IFD [CrawlThread]: now checkpoint "segments_1" [8 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [8 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_7.fnm" > > IFD [CrawlThread]: delete "_7.frq" > > IFD [CrawlThread]: delete "_7.prx" > > IFD [CrawlThread]: delete "_7.tis" > > IFD [CrawlThread]: delete "_7.tii" > > IFD [CrawlThread]: delete "_7.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 8 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 8 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=14.513 vs > > trigger=16 allocMB=16.808 vs trigger=16.8 byteBlockFree=0 > charBlockFree=0.75 > > IW 0 [CrawlThread]: DW: after free: freedMB=1.625 usedMB=14.513 > > allocMB=15.183 > > IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.057 allocMB=16.277 > > triggerMB=16 > > IW 0 [CrawlThread]: flush: segment=_8 docStoreSegment=_0 > > docStoreOffset=12015 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=3171 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _8 numDocs=3171 > > IW 0 [CrawlThread]: DW: oldRAMSize=16836608 newFlushedSize=7011110 > > docs/MB=474.252 new/old=41.642% > > IFD [CrawlThread]: now checkpoint "segments_1" [9 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [9 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_8.fnm" > > IFD [CrawlThread]: delete "_8.frq" > > IFD [CrawlThread]: delete "_8.prx" > > IFD [CrawlThread]: delete "_8.tis" > > IFD [CrawlThread]: delete "_8.tii" > > IFD [CrawlThread]: delete "_8.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 9 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 9 segments > > IW 0 [CrawlThread]: DW: RAM: now balance allocations: usedMB=15.678 vs > > trigger=16 allocMB=17.178 vs trigger=16.8 byteBlockFree=1.438 > > charBlockFree=0 > > IW 0 [CrawlThread]: DW: nothing to free > > IW 0 [CrawlThread]: DW: after free: freedMB=1.438 usedMB=15.678 > > allocMB=15.74 > > IW 0 [CrawlThread]: DW: RAM: now flush @ usedMB=16.067 allocMB=16.13 > > triggerMB=16 > > IW 0 [CrawlThread]: flush: segment=_9 docStoreSegment=_0 > > docStoreOffset=15186 flushDocs=true flushDeletes=false > flushDocStores=false > > numDocs=3401 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 > > IW 0 [CrawlThread]: DW: flush postings as segment _9 numDocs=3401 > > IW 0 [CrawlThread]: DW: oldRAMSize=16847872 newFlushedSize=6184635 > > docs/MB=576.624 new/old=36.709% > > IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = > > false] > > IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_9.fnm" > > IFD [CrawlThread]: delete "_9.frq" > > IFD [CrawlThread]: delete "_9.prx" > > IFD [CrawlThread]: delete "_9.tis" > > IFD [CrawlThread]: delete "_9.tii" > > IFD [CrawlThread]: delete "_9.nrm" > > IW 0 [CrawlThread]: LMP: findMerges: 10 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments > > IW 0 [CrawlThread]: LMP: 0 to 10: add this merge > > IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799- > >_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 _9:c3401->_0 [total 1 pending] > > IW 0 [CrawlThread]: DW: apply 0 buffered deleted terms and 1 deleted > docIDs > > and 0 deleted queries on 10 segments. > > IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = > > false] > > IW 0 [CrawlThread]: now flush at merge > > IW 0 [CrawlThread]: flush: segment=null docStoreSegment=_0 > > docStoreOffset=18587 flushDocs=false flushDeletes=false > flushDocStores=true > > numDocs=0 numBufDelTerms=0 > > IW 0 [CrawlThread]: index before flush _0:c2193->_0 _1:c1799->_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 _9:c3401->_0 > > IW 0 [CrawlThread]: flush shared docStore segment _0 > > IW 0 [CrawlThread]: DW: closeDocStore: 2 files to flush to segment _0 > > numDocs=18587 > > IFD [CrawlThread]: now checkpoint "segments_1" [10 segments ; isCommit = > > false] > > IFD [CrawlThread]: delete "_0.fdt" > > IFD [CrawlThread]: delete "_0.fdx" > > IFD [CrawlThread]: delete "_0.fdt" > > IFD [CrawlThread]: delete "_0.fdx" > > IW 0 [CrawlThread]: now merge > > merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _a > > merge=org.apache.lucene.index.mergepolicy$oneme...@b590b98 > > index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 > > IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 > > _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171- > >_0 > > _9:c3401->_0 into _a > > IW 0 [CrawlThread]: merge: total 18586 docs > > IW 0 [CrawlThread]: LMP: findMerges: 10 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments > > IW 0 [CrawlThread]: LMP: 0 to 10: add this merge > > IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799- > >_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 _9:c3401->_0 [total 1 pending] > > IW 0 [CrawlThread]: now merge > > merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _b > > merge=org.apache.lucene.index.mergepolicy$oneme...@b590658 > > index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 > > IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 > > _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171- > >_0 > > _9:c3401->_0 into _b > > IW 0 [CrawlThread]: merge: total 18586 docs > > IW 0 [CrawlThread]: LMP: findMerges: 10 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments > > IW 0 [CrawlThread]: LMP: 0 to 10: add this merge > > IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799- > >_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 _9:c3401->_0 [total 1 pending] > > IW 0 [CrawlThread]: now merge > > merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _c > > merge=org.apache.lucene.index.mergepolicy$oneme...@b5906b8 > > index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 > > IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 > > _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171- > >_0 > > _9:c3401->_0 into _c > > IW 0 [CrawlThread]: merge: total 18586 docs > > IW 0 [CrawlThread]: LMP: findMerges: 10 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments > > IW 0 [CrawlThread]: LMP: 0 to 10: add this merge > > IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799- > >_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 _9:c3401->_0 [total 1 pending] > > IW 0 [CrawlThread]: now merge > > merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _d > > merge=org.apache.lucene.index.mergepolicy$oneme...@b5906e8 > > index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 > > IW 0 [CrawlThread]: merging _0:c2193->_0 _1:c1799->_0 _2:c1013->_0 > > _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171- > >_0 > > _9:c3401->_0 into _d > > IW 0 [CrawlThread]: merge: total 18586 docs > > IW 0 [CrawlThread]: LMP: findMerges: 10 segments > > IW 0 [CrawlThread]: LMP: level 6.39909 to 7.14909: 10 segments > > IW 0 [CrawlThread]: LMP: 0 to 10: add this merge > > IW 0 [CrawlThread]: add merge to pendingMerges: _0:c2193->_0 _1:c1799- > >_0 > > _2:c1013->_0 _3:c427->_0 _4:c1145->_0 _5:c808->_0 _6:c1689->_0 _7:c2941- > >_0 > > _8:c3171->_0 _9:c3401->_0 [total 1 pending] > > IW 0 [CrawlThread]: now merge > > merge=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 into _e > > merge=org.apache.lucene.index.mergepolicy$oneme...@b590688 > > index=_0:c2193->_0 _1:c1799->_0 _2:c1013->_0 _3:c427->_0 _4:c1145->_0 > > _5:c808->_0 _6:c1689->_0 _7:c2941->_0 _8:c3171->_0 _9:c3401->_0 > > ....... > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
