Committed a fix.
On Thu, Jul 3, 2014 at 7:56 PM, Michael McCandless < [email protected]> wrote: > +1 to add the commit in RIW if it does an operation like forceMerge in > close. > > Mike McCandless > > http://blog.mikemccandless.com > > > On Thu, Jul 3, 2014 at 12:23 PM, Shai Erera <[email protected]> wrote: > > I am still digging - this isn't related to IW.changeCount, but > > SegInfos.counter. > > > > I think there's a bug in RandomIndexWriter - when you call close(), it > > randomly executes a forceMerge(), then closes, without committing. > > In IW.close(), we only check for lost changes prior to 5.0, therefore we > > don't hit the RuntimeException, and the changes made by forceMerge() are > > rolled-back. > > But at that point, _1.fdx was already written (even though it was > deleted by > > rollback()), therefore addIndexes cannot write it again. > > > > This seems to be a bug in RIW, not in core code, and if I add w.commit() > > after that randomForceMerge, the test passes. > > > > I will review it again later and commit, perhaps someone can take a > second > > look. > > > > Shai > > > > > > On Thu, Jul 3, 2014 at 6:50 PM, Shai Erera <[email protected]> wrote: > >> > >> It reproduces ... something's fishy with IW.changeCount -- seems that > >> after you open an IW on an existing Directory, changeCount = 0. I will > try > >> to reproduce this in a standalone test to verify if it's a general IW > bug or > >> not. > >> > >> Shai > >> > >> > >> On Thu, Jul 3, 2014 at 2:54 PM, Policeman Jenkins Server > >> <[email protected]> wrote: > >>> > >>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10711/ > >>> Java: 64bit/jdk1.8.0_20-ea-b15 -XX:+UseCompressedOops -XX:+UseG1GC > >>> > >>> 1 tests failed. > >>> FAILED: > >>> > org.apache.lucene.facet.taxonomy.OrdinalMappingReaderTest.testTaxonomyMergeUtils > >>> > >>> Error Message: > >>> file "_1.fdx" was already written to > >>> > >>> Stack Trace: > >>> java.io.IOException: file "_1.fdx" was already written to > >>> at > >>> > __randomizedtesting.SeedInfo.seed([2AFF1A4359AFD7EB:C8D30CECD284E330]:0) > >>> at > >>> > org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:492) > >>> at > >>> > org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) > >>> at > >>> > org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:110) > >>> at > >>> > org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) > >>> at > >>> > org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:327) > >>> at > >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) > >>> at > >>> org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2590) > >>> at > >>> > org.apache.lucene.facet.taxonomy.TaxonomyMergeUtils.merge(TaxonomyMergeUtils.java:56) > >>> at > >>> > org.apache.lucene.facet.taxonomy.OrdinalMappingReaderTest.testTaxonomyMergeUtils(OrdinalMappingReaderTest.java:73) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> at > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:483) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) > >>> at > >>> > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) > >>> at > >>> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) > >>> at > >>> > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) > >>> at > >>> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) > >>> at > >>> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >>> at > >>> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) > >>> at > >>> > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) > >>> at > >>> > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) > >>> at > >>> > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) > >>> at > >>> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) > >>> at > >>> > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >>> at > >>> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) > >>> at > >>> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) > >>> at > >>> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) > >>> at > >>> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) > >>> at > >>> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >>> at > >>> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) > >>> at java.lang.Thread.run(Thread.java:745) > >>> > >>> > >>> > >>> > >>> Build Log: > >>> [...truncated 8066 lines...] > >>> [junit4] Suite: > >>> org.apache.lucene.facet.taxonomy.OrdinalMappingReaderTest > >>> [junit4] 2> NOTE: reproduce with: ant test > >>> -Dtestcase=OrdinalMappingReaderTest > -Dtests.method=testTaxonomyMergeUtils > >>> -Dtests.seed=2AFF1A4359AFD7EB -Dtests.multiplier=3 -Dtests.slow=true > >>> -Dtests.locale=hi_IN -Dtests.timezone=SystemV/MST7MDT > >>> -Dtests.file.encoding=UTF-8 > >>> [junit4] ERROR 0.54s J1 | > >>> OrdinalMappingReaderTest.testTaxonomyMergeUtils <<< > >>> [junit4] > Throwable #1: java.io.IOException: file "_1.fdx" was > >>> already written to > >>> [junit4] > at > >>> > __randomizedtesting.SeedInfo.seed([2AFF1A4359AFD7EB:C8D30CECD284E330]:0) > >>> [junit4] > at > >>> > org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:492) > >>> [junit4] > at > >>> > org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44) > >>> [junit4] > at > >>> > org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:110) > >>> [junit4] > at > >>> > org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120) > >>> [junit4] > at > >>> > org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:327) > >>> [junit4] > at > >>> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) > >>> [junit4] > at > >>> org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2590) > >>> [junit4] > at > >>> > org.apache.lucene.facet.taxonomy.TaxonomyMergeUtils.merge(TaxonomyMergeUtils.java:56) > >>> [junit4] > at > >>> > org.apache.lucene.facet.taxonomy.OrdinalMappingReaderTest.testTaxonomyMergeUtils(OrdinalMappingReaderTest.java:73) > >>> [junit4] > at java.lang.Thread.run(Thread.java:745) > >>> [junit4] 2> NOTE: test params are: codec=Lucene41, > >>> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, > locale=hi_IN, > >>> timezone=SystemV/MST7MDT > >>> [junit4] 2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation > >>> 1.8.0_20-ea (64-bit)/cpus=8,threads=1,free=105436336,total=132120576 > >>> [junit4] 2> NOTE: All tests run in this JVM: > >>> [TestTaxonomyFacetCounts, OrdinalMappingReaderTest] > >>> [junit4] Completed on J1 in 0.59s, 1 test, 1 error <<< FAILURES! > >>> > >>> [...truncated 70 lines...] > >>> BUILD FAILED > >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The > >>> following error occurred while executing this line: > >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The > >>> following error occurred while executing this line: > >>> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The > >>> following error occurred while executing this line: > >>> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: > >>> The following error occurred while executing this line: > >>> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544: > >>> The following error occurred while executing this line: > >>> > >>> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019: > >>> The following error occurred while executing this line: > >>> > >>> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: > >>> The following error occurred while executing this line: > >>> > >>> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: > >>> The following error occurred while executing this line: > >>> > >>> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: > >>> There were test failures: 23 suites, 155 tests, 1 error, 1 ignored (1 > >>> assumption) > >>> > >>> Total time: 30 minutes 21 seconds > >>> Build step 'Invoke Ant' marked build as failure > >>> [description-setter] Description set: Java: 64bit/jdk1.8.0_20-ea-b15 > >>> -XX:+UseCompressedOops -XX:+UseG1GC > >>> Archiving artifacts > >>> Recording test results > >>> Email was triggered for: Failure - Any > >>> Sending email for trigger: Failure - Any > >>> > >>> > >>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
