[
https://issues.apache.org/jira/browse/LUCENE-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14392117#comment-14392117
]
Shai Erera commented on LUCENE-6386:
------------------------------------
So if we have the index in the following state, according to what you describe
we will use additional 3X space:
{noformat}
source: [s1 s2 s3] [s4 s5 s6] [s7 s8 s9]
level1 (not CFS'd): [s10] [s11] [s12]
level2 (non CFS'd): [s13]
result (CFS'd): [s14]
{noformat}
When we create level2, we indeed use additional 2X space. But is there a reason
why when that's done we don't delete level1's intermediate files? Nobody
references them correct?
If we don't, then what happens when that forceMerge() hits 10 levels of merges?
Will all the intermediate files be kept in the index? If so, then during
forceMerge the additional space required can be much higher than just 3X.
> TestIndexWriterForceMerge still unreliable in NIGHTLY
> -----------------------------------------------------
>
> Key: LUCENE-6386
> URL: https://issues.apache.org/jira/browse/LUCENE-6386
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Robert Muir
> Assignee: Michael McCandless
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-6386.patch
>
>
> Discovered by ryan beasting (trunk):
> ant test -Dtestcase=TestIndexWriterForceMerge
> -Dtests.method=testForceMergeTempSpaceUsage -Dtests.seed=DC9ADB74850A581B
> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.locale=sr__#Latn -Dtests.timezone=Indian/Chagos -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> {noformat}
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestIndexWriterForceMerge
> -Dtests.method=testForceMergeTempSpaceUsage -Dtests.seed=DC9ADB74850A581B
> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.locale=sr__#Latn -Dtests.timezone=Indian/Chagos -Dtests.asserts=true
> -Dtests.file.encoding=US-ASCII
> [junit4] FAILURE 1.20s |
> TestIndexWriterForceMerge.testForceMergeTempSpaceUsage <<<
> [junit4] > Throwable #1: java.lang.AssertionError: forceMerge used too
> much temporary space: starting usage was 291570 bytes; final usage was 262469
> bytes; max temp usage was 1079501 but should have been 874710 (= 3X starting
> usage), BEFORE=
> [junit4] > _u.scf 146329
> [junit4] > _u.si 635
> [junit4] > |- (inside compound file) _u.fld 2214
> [junit4] > |- (inside compound file) _u.inf 392
> [junit4] > |- (inside compound file) _u.len 2381
> [junit4] > |- (inside compound file) _u.pst 36758
> [junit4] > |- (inside compound file) _u.vec 104144
> [junit4] > _s.pst 1338
> [junit4] > _s.inf 392
> [junit4] > _s.fld 94
> [junit4] > _s.len 221
> [junit4] > _s.vec 3744
> [junit4] > _s.si 624
> [junit4] > _t.fld 94
> [junit4] > _t.len 221
> [junit4] > _t.pst 1338
> [junit4] > _t.inf 392
> [junit4] > _t.vec 3744
> [junit4] > _t.si 624
> [junit4] > _v.fld 94
> [junit4] > _v.pst 1338
> [junit4] > _v.inf 392
> [junit4] > _v.vec 3744
> [junit4] > _v.si 624
> [junit4] > _v.len 221
> [junit4] > _w.len 221
> [junit4] > _w.pst 1338
> [junit4] > _w.inf 392
> [junit4] > _w.fld 94
> [junit4] > _w.si 624
> [junit4] > _w.vec 3744
> [junit4] > _x.vec 3744
> [junit4] > _x.inf 392
> [junit4] > _x.pst 1338
> [junit4] > _x.fld 94
> [junit4] > _x.si 624
> [junit4] > _x.len 221
> [junit4] > _y.fld 94
> [junit4] > _y.pst 1338
> [junit4] > _y.inf 392
> [junit4] > _y.si 624
> [junit4] > _y.vec 3744
> [junit4] > _y.len 221
> [junit4] > _z.fld 94
> [junit4] > _z.pst 1338
> [junit4] > _z.inf 392
> [junit4] > _z.len 221
> [junit4] > _z.vec 3744
> [junit4] > _z.si 624
> [junit4] > _10.si 630
> [junit4] > _10.fld 94
> [junit4] > _10.pst 1338
> [junit4] > _10.inf 392
> [junit4] > _10.vec 3744
> [junit4] > _10.len 221
> [junit4] > _11.len 221
> [junit4] > _11.si 630
> [junit4] > _11.vec 3744
> [junit4] > _11.pst 1338
> [junit4] > _11.inf 392
> [junit4] > _11.fld 94
> [junit4] > _12.vec 3744
> [junit4] > _12.si 630
> [junit4] > _12.len 221
> [junit4] > _12.fld 94
> [junit4] > _12.pst 1338
> [junit4] > _12.inf 392
> [junit4] > _13.fld 94
> [junit4] > _13.vec 3744
> [junit4] > _13.si 630
> [junit4] > _13.pst 1338
> [junit4] > _13.inf 392
> [junit4] > _13.len 221
> [junit4] > _14.fld 94
> [junit4] > _14.pst 1338
> [junit4] > _14.inf 392
> [junit4] > _14.si 630
> [junit4] > _14.vec 3744
> [junit4] > _14.len 221
> [junit4] > _15.len 221
> [junit4] > _15.vec 3744
> [junit4] > _15.si 630
> [junit4] > _15.pst 1338
> [junit4] > _15.inf 392
> [junit4] > _15.fld 94
> [junit4] > _16.vec 3744
> [junit4] > _16.len 221
> [junit4] > _16.fld 94
> [junit4] > _16.si 630
> [junit4] > _16.pst 1338
> [junit4] > _16.inf 392
> [junit4] > _17.vec 3744
> [junit4] > _17.pst 1338
> [junit4] > _17.inf 392
> [junit4] > _17.len 221
> [junit4] > _17.si 630
> [junit4] > _17.fld 94
> [junit4] > _18.pst 1338
> [junit4] > _18.inf 392
> [junit4] > _18.len 221
> [junit4] > _18.vec 3744
> [junit4] > _18.si 630
> [junit4] > _18.fld 94
> [junit4] > _19.fld 94
> [junit4] > _19.si 630
> [junit4] > _19.len 221
> [junit4] > _19.vec 3744
> [junit4] > _19.pst 1338
> [junit4] > _19.inf 392
> [junit4] > _1a.fld 94
> [junit4] > _1a.pst 1338
> [junit4] > _1a.inf 392
> [junit4] > _1a.len 221
> [junit4] > _1a.vec 3744
> [junit4] > _1a.si 630
> [junit4] > _1b.fld 94
> [junit4] > _1b.si 630
> [junit4] > _1b.pst 1338
> [junit4] > _1b.inf 392
> [junit4] > _1b.vec 3744
> [junit4] > _1b.len 221
> [junit4] > _1c.vec 3744
> [junit4] > _1c.pst 1338
> [junit4] > _1c.inf 392
> [junit4] > _1c.fld 94
> [junit4] > _1c.len 221
> [junit4] > _1c.si 630
> [junit4] > _1d.pst 1338
> [junit4] > _1d.inf 392
> [junit4] > _1d.fld 94
> [junit4] > _1d.len 221
> [junit4] > _1d.vec 3744
> [junit4] > _1d.si 630
> [junit4] > _1e.fld 94
> [junit4] > _1e.pst 1338
> [junit4] > _1e.inf 392
> [junit4] > _1e.len 221
> [junit4] > _1e.vec 3744
> [junit4] > _1e.si 630
> [junit4] > _1f.fld 40
> [junit4] > _1f.pst 195
> [junit4] > _1f.inf 392
> [junit4] > _1f.len 149
> [junit4] > _1f.vec 405
> [junit4] > _1f.si 629
> [junit4] > AFTER=
> [junit4] > _1j.si 635
> [junit4] > _1j.scf 261701
> [junit4] > |- (inside compound file) _1j.fld 3982
> [junit4] > |- (inside compound file) _1j.inf 392
> [junit4] > |- (inside compound file) _1j.len 4149
> [junit4] > |- (inside compound file) _1j.pst 66151
> [junit4] > |- (inside compound file) _1j.vec 186577
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([DC9ADB74850A581B:C6581887EB1ABBDC]:0)
> [junit4] > at
> org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage(TestIndexWriterForceMerge.java:199)
> [junit4] > at java.lang.Thread.run(Thread.java:745)
> [junit4] 2> NOTE: test params are: codec=SimpleText,
> sim=RandomSimilarityProvider(queryNorm=false,coord=no): {id=IB SPL-D2,
> content=DFR I(F)B1}, locale=sr__#Latn, timezone=Indian/Chagos
> [junit4] 2> NOTE: Linux 3.13.0-45-generic amd64/Oracle Corporation
> 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=208092000,total=253231104
> [junit4] 2> NOTE: All tests run in this JVM: [TestIndexWriterForceMerge]
> [junit4] Completed in 1.54s, 1 test, 1 failure <<< FAILURES!
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
