Simon Willnauer created LUCENE-8243:
---------------------------------------
Summary: IndexWriter might delete DV update files if addIndices
are invovled
Key: LUCENE-8243
URL: https://issues.apache.org/jira/browse/LUCENE-8243
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 7.4, master (8.0)
Reporter: Simon Willnauer
Fix For: 7.4, master (8.0)
Attachments: broken_dv_update.patch
the attached test fails with this output:
{noformat}
/Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home/bin/java -ea
-Djava.security.egd=file:/dev/./urandom -Didea.test.cyclic.buffer.size=1048576
-Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ
IDEA.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ
IDEA.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ
IDEA.app/Contents/plugins/junit/lib/junit5-rt.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/test:/Users/simonw/projects/lucene-solr/idea-build/lucene/test-framework/classes/java:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/junit-4.10.jar:/Users/simonw/projects/lucene-solr/lucene/test-framework/lib/randomizedtesting-runner-2.5.3.jar:/Users/simonw/projects/lucene-solr/idea-build/lucene/codecs/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/java:/Users/simonw/projects/lucene-solr/idea-build/lucene/core/classes/test"
com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4
org.apache.lucene.index.TestAddIndexes,testAddIndexesDVUpdate
IFD 0 [2018-04-06T19:27:27.176036Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init:
current segments file is "segments_1";
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@27cf18f0
IFD 0 [2018-04-06T19:27:27.188066Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init: load
commit "segments_1"
IFD 0 [2018-04-06T19:27:27.189800Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init:
seg=_0 set nextWriteDelGen=2 vs current=1
IFD 0 [2018-04-06T19:27:27.190053Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init:
removing unreferenced file "_0_1_Lucene70_0.dvd"
IFD 0 [2018-04-06T19:27:27.190224Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init:
removing unreferenced file "_0_1.fnm"
IFD 0 [2018-04-06T19:27:27.190371Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init:
removing unreferenced file "_0_1_Lucene70_0.dvm"
IFD 0 [2018-04-06T19:27:27.190528Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: delete
[_0_1_Lucene70_0.dvd, _0_1.fnm, _0_1_Lucene70_0.dvm]
IFD 0 [2018-04-06T19:27:27.192558Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now
checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit =
false]
IFD 0 [2018-04-06T19:27:27.192806Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec to
checkpoint
IW 0 [2018-04-06T19:27:27.193012Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: init:
create=false
IW 0 [2018-04-06T19:27:27.193428Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
dir=MockDirectoryWrapper(RAMDirectory@79d3c690
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b)
index=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
version=8.0.0
analyzer=org.apache.lucene.analysis.MockAnalyzer
ramBufferSizeMB=16.0
maxBufferedDocs=503
mergedSegmentWarmer=null
delPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
commit=null
openMode=CREATE_OR_APPEND
similarity=org.apache.lucene.search.similarities.AssertingSimilarity
mergeScheduler=org.apache.lucene.index.SerialMergeScheduler@2f3feff6
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=8, maxDocsPerChunk=6, blockSize=201),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=8, blockSize=201))
infoStream=org.apache.lucene.util.PrintStreamInfoStream
mergePolicy=[TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=44,
maxMergedSegmentMB=6.255859375, floorSegmentMB=0.38671875,
forceMergeDeletesPctAllowed=4.456652110760543, segmentsPerTier=31.0,
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.877330376985384
indexerThreadPool=org.apache.lucene.index.DocumentsWriterPerThreadPool@257ebcdb
readerPooling=true
perThreadHardLimitMB=1945
useCompoundFile=true
commitOnClose=true
indexSort=null
checkPendingFlushOnUpdate=true
softDeletesField=null
writer=org.apache.lucene.index.IndexWriter@17b77ed5
IW 0 [2018-04-06T19:27:27.194085Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
MMapDirectory.UNMAP_SUPPORTED=true
IW 0 [2018-04-06T19:27:27.194347Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now flush
at close
IW 0 [2018-04-06T19:27:27.194630Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: start
flush: applyAllDeletes=true
IW 0 [2018-04-06T19:27:27.194864Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: index
before flush _0(8.0.0):C1:fieldInfosGen=1:dvGen=1
DW 0 [2018-04-06T19:27:27.194964Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
startFullFlush
DW 0 [2018-04-06T19:27:27.195096Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]
finishFullFlush success=true
IW 0 [2018-04-06T19:27:27.195181Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now apply
all deletes for all segments buffered updates bytesUsed=0 reader pool
bytesUsed=0
BD 0 [2018-04-06T19:27:27.195260Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: waitApply:
no deletes to apply
IW 0 [2018-04-06T19:27:27.195330Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
waitForMerges
IW 0 [2018-04-06T19:27:27.195385Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
waitForMerges done
IW 0 [2018-04-06T19:27:27.195448Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit:
start
IW 0 [2018-04-06T19:27:27.195504Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit:
enter lock
IW 0 [2018-04-06T19:27:27.195560Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit:
now prepare
IW 0 [2018-04-06T19:27:27.195779Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
prepareCommit: flush
IW 0 [2018-04-06T19:27:27.195873Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: index
before flush _0(8.0.0):C1:fieldInfosGen=1:dvGen=1
DW 0 [2018-04-06T19:27:27.195954Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
startFullFlush
IW 0 [2018-04-06T19:27:27.196076Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now apply
all deletes for all segments buffered updates bytesUsed=0 reader pool
bytesUsed=0
BD 0 [2018-04-06T19:27:27.196170Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: waitApply:
no deletes to apply
DW 0 [2018-04-06T19:27:27.196299Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]
finishFullFlush success=true
IW 0 [2018-04-06T19:27:27.196382Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]:
startCommit(): start
IW 0 [2018-04-06T19:27:27.196455Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: skip
startCommit(): no changes pending
IW 0 [2018-04-06T19:27:27.196542Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit:
pendingCommit == null; skip
IW 0 [2018-04-06T19:27:27.196717Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit:
took 0.8 msec
IW 0 [2018-04-06T19:27:27.196791Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: commit:
done
IW 0 [2018-04-06T19:27:27.196950Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: rollback
IW 0 [2018-04-06T19:27:27.197032Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: all
running merges have aborted
IW 0 [2018-04-06T19:27:27.197326Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: rollback:
done finish merges
DW 0 [2018-04-06T19:27:27.197411Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: abort
DW 0 [2018-04-06T19:27:27.197486Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: done abort
success=true
IW 0 [2018-04-06T19:27:27.197605Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: rollback:
infos=_0(8.0.0):C1:fieldInfosGen=1:dvGen=1
IFD 0 [2018-04-06T19:27:27.197706Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: now
checkpoint "_0(8.0.0):C1:fieldInfosGen=1:dvGen=1" [1 segments ; isCommit =
false]
IFD 0 [2018-04-06T19:27:27.197837Z;
TEST-TestAddIndexes.testAddIndexesDVUpdate-seed#[9F04EE6B720B6BFD]]: 0 msec to
checkpoint
java.nio.file.NoSuchFileException: _0_1.fnm in dir=RAMDirectory@79d3c690
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@795a0c8b
at
__randomizedtesting.SeedInfo.seed([9F04EE6B720B6BFD:E6EC23304E67E247]:0)
at
org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:750)
at
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
at
org.apache.lucene.store.MockDirectoryWrapper.openChecksumInput(MockDirectoryWrapper.java:1072)
at
org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
at
org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1207)
at
org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1227)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1114)
at
org.apache.lucene.index.TestAddIndexes.testAddIndexesDVUpdate(TestAddIndexes.java:1364)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
-Dtests.method=testAddIndexesDVUpdate -Dtests.seed=9F04EE6B720B6BFD
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-HN
-Dtests.timezone=Asia/Singapore -Dtests.asserts=true -Dtests.file.encoding=UTF-8
NOTE: test params are:
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=8, maxDocsPerChunk=6, blockSize=201),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=8, blockSize=201)),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@55f3c81b),
locale=es-HN, timezone=Asia/Singapore
NOTE: Mac OS X 10.13.3 x86_64/Oracle Corporation 9.0.1
(64-bit)/cpus=4,threads=1,free=231833088,total=268435456
NOTE: All tests run in this JVM: [TestAddIndexes]
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
