I committed a fix. But I think memoryPF can get really wasteful here. This is ultimately the same problem as the TestDuelingCodecs OOM.
On Tue, Mar 31, 2015 at 11:08 PM, Robert Muir <[email protected]> wrote: > This reproduces. I'm digging. > > On Mon, Mar 2, 2015 at 9:52 AM, Apache Jenkins Server > <[email protected]> wrote: >> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/774/ >> >> 1 tests failed. >> REGRESSION: >> org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage >> >> Error Message: >> forceMerge used too much temporary space: starting usage was 379542 bytes; >> final usage was 442916 bytes; max temp usage was 1669519 but should have >> been 1328748 (= 3X starting usage) >> >> Stack Trace: >> java.lang.AssertionError: forceMerge used too much temporary space: starting >> usage was 379542 bytes; final usage was 442916 bytes; max temp usage was >> 1669519 but should have been 1328748 (= 3X starting usage) >> at >> __randomizedtesting.SeedInfo.seed([AD6008DD6F02F612:B7A2CB2E011215D5]:0) >> at org.junit.Assert.fail(Assert.java:93) >> at org.junit.Assert.assertTrue(Assert.java:43) >> at >> org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage(TestIndexWriterForceMerge.java:181) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) >> at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> >> Build Log: >> [...truncated 999 lines...] >> [junit4] Suite: org.apache.lucene.index.TestIndexWriterForceMerge >> [junit4] 2> NOTE: download the large Jenkins line-docs file by running >> 'ant get-jenkins-line-docs' in the lucene directory. >> [junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestIndexWriterForceMerge >> -Dtests.method=testForceMergeTempSpaceUsage -Dtests.seed=AD6008DD6F02F612 >> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true >> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt >> -Dtests.locale=en_GB -Dtests.timezone=Pacific/Midway -Dtests.asserts=true >> -Dtests.file.encoding=UTF-8 >> [junit4] FAILURE 2.11s J1 | >> TestIndexWriterForceMerge.testForceMergeTempSpaceUsage <<< >> [junit4] > Throwable #1: java.lang.AssertionError: forceMerge used too >> much temporary space: starting usage was 379542 bytes; final usage was >> 442916 bytes; max temp usage was 1669519 but should have been 1328748 (= 3X >> starting usage) >> [junit4] > at >> __randomizedtesting.SeedInfo.seed([AD6008DD6F02F612:B7A2CB2E011215D5]:0) >> [junit4] > at >> org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage(TestIndexWriterForceMerge.java:181) >> [junit4] > at java.lang.Thread.run(Thread.java:745) >> [junit4] 2> NOTE: leaving temporary files on disk at: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J1/temp/lucene.index.TestIndexWriterForceMerge >> AD6008DD6F02F612-001 >> [junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): >> {field=PostingsFormat(name=LuceneFixedGap), id=Lucene50(blocksize=128), >> content=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, >> sim=DefaultSimilarity, locale=en_GB, timezone=Pacific/Midway >> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >> 1.7.0_65 (64-bit)/cpus=16,threads=1,free=186424664,total=358088704 >> [junit4] 2> NOTE: All tests run in this JVM: >> [TestDocInverterPerFieldErrorInfo, TestStressIndexing, TestNumericUtils, >> TestLockFactory, TestFilteredSearch, TestDocCount, TestIsCurrent, >> TestConsistentFieldNumbers, TestSimpleFSDirectory, >> TestAssertingStoredFieldsFormat, Nested, Nested, >> TestParallelReaderEmptyIndex, TestDateFilter, TestQueryBuilder, >> TestCheckIndex, TestBufferedChecksum, TestTermVectors, TestIndexFileDeleter, >> TestLiveFieldValues, TestIndexWriterDelete, TestTopDocsCollector, >> TestMultiMMap, TestDocValuesScoring, TestFuzzyQuery, >> TestLucene50StoredFieldsFormat, TestStressAdvance, TestIndexSearcher, >> TestPerFieldDocValuesFormat, TestNumericTokenStream, >> TestPackedTokenAttributeImpl, TestManyFields, TestNumericDocValuesUpdates, >> TestGraphTokenizers, Test2BPagedBytes, TestTimSorter, TestEliasFanoDocIdSet, >> TestInfoStream, TestBlockPostingsFormat3, TestRegexpQuery, >> TestDocValuesRangeQuery, TestDocTermOrdsRewriteMethod, >> TestSloppyPhraseQuery2, TestMockDirectoryWrapper, TestBlockPostingsFormat, >> TestCharsRefBuilder, TestIndexWriterMergePolicy, TestBooleanQuery, >> TestMinShouldMatch2, TestTerm, TestIntsRef, TestAssertions, >> TestFieldMaskingSpanQuery, TestPostingsOffsets, TestArrayUtil, >> TestLongPostings, TestIndexWriterForceMerge] >> [junit4] Completed on J1 in 6.51s, 4 tests, 1 failure <<< FAILURES! >> >> [...truncated 663 lines...] >> BUILD FAILED >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:49: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1370: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:976: >> There were test failures: 421 suites, 3373 tests, 1 failure, 40 ignored (30 >> assumptions) >> >> Total time: 48 minutes 55 seconds >> Build step 'Invoke Ant' marked build as failure >> Archiving artifacts >> Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #716 >> Archived 5 artifacts >> Archive block size is 32768 >> Received 0 blocks and 3647849 bytes >> Compression is 0.0% >> Took 1.1 sec >> Recording test results >> Email was triggered for: Failure >> Sending email for trigger: Failure >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
