I found the cause: there is an off-by-one error in the decoder (Lucene54DocValuesProducer) that triggers such an error when the number of docs is a multiple of 65536 minus one. I'll work on a fix and tests. We'll have to do a 5.4.1 for this I'm afraid.
Le lun. 11 janv. 2016 à 09:27, Adrien Grand <[email protected]> a écrit : > I'll dig. > > Le lun. 11 janv. 2016 à 02:28, Michael McCandless < > [email protected]> a écrit : > >> It reproduces! I haven't tried to dig ... >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >> >> On Sun, Jan 10, 2016 at 9:58 AM, Policeman Jenkins Server >> <[email protected]> wrote: >> > Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/15210/ >> > Java: 64bit/jdk1.8.0_66 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC >> > >> > 1 tests failed. >> > FAILED: >> org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat.testSparseDocValuesVsStoredFields >> > >> > Error Message: >> > Invalid vInt detected (too many bits) >> > >> > Stack Trace: >> > java.io.IOException: Invalid vInt detected (too many bits) >> > at >> __randomizedtesting.SeedInfo.seed([8AAEA8F6A2842A79:DE7574E558080C36]:0) >> > at >> org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) >> > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.readNumericEntry(Lucene54DocValuesProducer.java:355) >> > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.readFields(Lucene54DocValuesProducer.java:243) >> > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:122) >> > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) >> > at >> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:268) >> > at >> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:358) >> > at >> org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) >> > at >> org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) >> > at >> org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:147) >> > at >> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:81) >> > at >> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) >> > at >> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50) >> > at >> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731) >> > at >> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:73) >> > at >> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) >> > at >> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:375) >> > at >> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:305) >> > at >> org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat.doTestSparseDocValuesVsStoredFields(TestLucene54DocValuesFormat.java:198) >> > at >> org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat.testSparseDocValuesVsStoredFields(TestLucene54DocValuesFormat.java:144) >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:497) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921) >> > at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> > at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> > at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >> > at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> > at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816) >> > at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827) >> > at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) >> > at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> > at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> > at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >> > at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> > at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) >> > at java.lang.Thread.run(Thread.java:745) >> > Suppressed: org.apache.lucene.index.CorruptIndexException: >> checksum passed (77cdfc0e). possibly transient resource issue, or a Lucene >> or JVM bug >> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/core/test/J1/temp/lucene.codecs.lucene54.TestLucene54DocValuesFormat_8AAEA8F6A2842A79-001/tempDir-004/_k8_Lucene54_0.dvm")))) >> > at >> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:380) >> > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:126) >> > ... 52 more >> > >> > >> > >> > >> > Build Log: >> > [...truncated 1801 lines...] >> > [junit4] Suite: >> org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat >> > [junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestLucene54DocValuesFormat >> -Dtests.method=testSparseDocValuesVsStoredFields >> -Dtests.seed=8AAEA8F6A2842A79 -Dtests.multiplier=3 -Dtests.slow=true >> -Dtests.locale=ga -Dtests.timezone=Europe/Dublin -Dtests.asserts=true >> -Dtests.file.encoding=UTF-8 >> > [junit4] ERROR 160s J1 | >> TestLucene54DocValuesFormat.testSparseDocValuesVsStoredFields <<< >> > [junit4] > Throwable #1: java.io.IOException: Invalid vInt >> detected (too many bits) >> > [junit4] > at >> __randomizedtesting.SeedInfo.seed([8AAEA8F6A2842A79:DE7574E558080C36]:0) >> > [junit4] > at >> org.apache.lucene.store.DataInput.readVInt(DataInput.java:141) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.readNumericEntry(Lucene54DocValuesProducer.java:355) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.readFields(Lucene54DocValuesProducer.java:243) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:122) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesFormat.fieldsProducer(Lucene54DocValuesFormat.java:113) >> > [junit4] > at >> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:268) >> > [junit4] > at >> org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:358) >> > [junit4] > at >> org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:51) >> > [junit4] > at >> org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:67) >> > [junit4] > at >> org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:147) >> > [junit4] > at >> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:81) >> > [junit4] > at >> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) >> > [junit4] > at >> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:50) >> > [junit4] > at >> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731) >> > [junit4] > at >> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:73) >> > [junit4] > at >> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) >> > [junit4] > at >> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:375) >> > [junit4] > at >> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:305) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat.doTestSparseDocValuesVsStoredFields(TestLucene54DocValuesFormat.java:198) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.TestLucene54DocValuesFormat.testSparseDocValuesVsStoredFields(TestLucene54DocValuesFormat.java:144) >> > [junit4] > at java.lang.Thread.run(Thread.java:745) >> > [junit4] > Suppressed: >> org.apache.lucene.index.CorruptIndexException: checksum passed (77cdfc0e). >> possibly transient resource issue, or a Lucene or JVM bug >> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(NIOFSIndexInput(path="/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/core/test/J1/temp/lucene.codecs.lucene54.TestLucene54DocValuesFormat_8AAEA8F6A2842A79-001/tempDir-004/_k8_Lucene54_0.dvm")))) >> > [junit4] > at >> org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:380) >> > [junit4] > at >> org.apache.lucene.codecs.lucene54.Lucene54DocValuesProducer.<init>(Lucene54DocValuesProducer.java:126) >> > [junit4] > ... 52 more >> > [junit4] IGNOR/A 0.00s J1 | >> TestLucene54DocValuesFormat.testSortedSetVariableLengthManyVsStoredFields >> > [junit4] > Assumption #1: 'nightly' test group is disabled >> (@Nightly()) >> > [junit4] IGNOR/A 0.00s J1 | >> TestLucene54DocValuesFormat.testTermsEnumRandomMany >> > [junit4] > Assumption #1: 'nightly' test group is disabled >> (@Nightly()) >> > [junit4] IGNOR/A 0.00s J1 | >> TestLucene54DocValuesFormat.testSortedVariableLengthManyVsStoredFields >> > [junit4] > Assumption #1: 'nightly' test group is disabled >> (@Nightly()) >> > [junit4] 2> NOTE: leaving temporary files on disk at: >> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/core/test/J1/temp/lucene.codecs.lucene54.TestLucene54DocValuesFormat_8AAEA8F6A2842A79-001 >> > [junit4] 2> NOTE: test params are: >> codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, >> chunkSize=8, maxDocsPerChunk=773, blockSize=531), >> termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, >> chunkSize=8, blockSize=531)), >> sim=RandomSimilarityProvider(queryNorm=true,coord=no): {fieldname=DFR >> I(ne)L2, docId=DFR I(F)3(800.0), id=DFR I(F)1}, locale=ga, >> timezone=Europe/Dublin >> > [junit4] 2> NOTE: Linux 3.19.0-42-generic amd64/Oracle Corporation >> 1.8.0_66 (64-bit)/cpus=12,threads=1,free=376607504,total=518979584 >> > [junit4] 2> NOTE: All tests run in this JVM: [TestPrefixQuery, >> TestWeakIdentityMap, TestPackedInts, TestNewestSegment, TestIntBlockPool, >> TestSingleInstanceLockFactory, TestFieldValueQuery, TestSearchAfter, >> TestAllFilesHaveChecksumFooter, TestNumericDocValuesUpdates, >> TestFastDecompressionMode, TestWildcardRandom, TestFilteredSearch, >> TestLucene50TermVectorsFormat, TestIndexWriterForceMerge, TestFlex, >> TestBooleanQueryVisitSubscorers, TestUniqueTermCount, TestCrash, >> TestDemoParallelLeafReader, TestManyFields, TestStringHelper, >> TestIndexWriterLockRelease, TestReaderWrapperDVTypeCheck, >> TestIndexWriterExceptions, TestByteSlices, TestDisjunctionMaxQuery, >> TestRateLimiter, TestMultiCollector, TestDirectoryReaderReopen, >> TestSpansAdvanced2, TestQueryBuilder, TestScorerPerf, >> TestFieldMaskingSpanQuery, TestMultiTermQueryRewrites, TestDateSort, >> TestMergedIterator, TestPackedTokenAttributeImpl, TestIndexInput, >> TestSegmentTermDocs, TestLRUQueryCache, TestBooleanMinShouldMatch, >> TestDirectMonotonic, TestInPlaceMergeSorter, TestDocument, >> TestAllFilesHaveCodecHeader, TestIndexWriterThreadsToSegments, >> TestTermsEnum2, TestBoostQuery, TestTermRangeFilter, TestPayloads, >> TestDocIdSetBuilder, TestCodecs, TestStressAdvance, TestFieldReuse, >> TestBlockPostingsFormat, TestTermVectorsReader, TestFixedBitSet, >> TestTermVectors, TestStressIndexing2, TestPrefixRandom, TestDateTools, >> TestRecyclingByteBlockAllocator, TestTermdocPerf, TestTermRangeQuery, >> TestSortedSetSelector, Test2BNumericDocValues, TestIsCurrent, >> TestPositiveScoresOnlyCollector, TestAddIndexes, TestMultiMMap, >> TestSearcherManager, TestBasics, TestIndexWriterDelete, TestNRTThreads, >> TestBoolean2, TestTimeLimitingCollector, TestHighCompressionMode, >> TestTopDocsMerge, TestStressIndexing, TestMinimize, TestRollingUpdates, >> TestDocTermOrdsRewriteMethod, TestNRTReaderWithThreads, >> TestLevenshteinAutomata, TestLockFactory, TestSegmentMerger, >> TestPhraseQuery, TestRegexpRandom2, TestNRTCachingDirectory, >> TestFieldCacheRewriteMethod, TestTransactions, TestMultiThreadTermVectors, >> TestPerFieldPostingsFormat2, TestMultiDocValues, TestFieldsReader, >> TestDocValuesIndexing, TestNumericUtils, TestIndexWriterOnDiskFull, >> TestCustomNorms, TestMixedCodecs, TestMergeSchedulerExternal, >> TestTermVectorsWriter, TestOmitPositions, TestParallelLeafReader, >> TestThreadedForceMerge, TestStressNRT, TestForceMergeForever, >> TestNearSpansOrdered, TestMultiFields, TestDocumentWriter, >> TestMultiTermConstantScore, TestWildcard, TestSimilarity2, >> TestExternalCodecs, TestNoDeletionPolicy, TestPerSegmentDeletes, >> TestPrefixInBooleanQuery, TestBooleanQuery, TestUnicodeUtil, >> TestAutomatonQuery, TestFileSwitchDirectory, TestCheckIndex, >> TestSizeBoundedForceMerge, TestCompiledAutomaton, TestParallelTermEnum, >> TestElevationComparator, TestConjunctions, >> TestSpanExplanationsOfNonMatches, TestSmallFloat, TestNumericTokenStream, >> TestFilterLeafReader, TestCachingTokenFilter, Test2BPostings, >> TestBinaryDocument, TestBooleanScorer, TestIndexWriterOnJRECrash, >> TestVersion, TestReaderClosed, TestScoreCachingWrappingScorer, >> TestIndexCommit, Test2BTerms, Test2BBinaryDocValues, >> TestBlockPostingsFormat2, TestLucene50StoredFieldsFormatHighCompression, >> TestLucene54DocValuesFormat] >> > [junit4] Completed [415/415 (1!)] on J1 in 539.47s, 107 tests, 1 >> error, 3 skipped <<< FAILURES! >> > >> > [...truncated 11 lines...] >> > BUILD FAILED >> > /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:794: The >> following error occurred while executing this line: >> > /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:738: The >> following error occurred while executing this line: >> > /home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:59: The >> following error occurred while executing this line: >> > /home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:50: The >> following error occurred while executing this line: >> > >> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1444: >> The following error occurred while executing this line: >> > >> /home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1000: >> There were test failures: 415 suites (1 ignored), 3349 tests, 1 error, 52 >> ignored (48 assumptions) [seed: 8AAEA8F6A2842A79] >> > >> > Total time: 13 minutes 4 seconds >> > Build step 'Invoke Ant' marked build as failure >> > Archiving artifacts >> > [WARNINGS] Skipping publisher since build result is FAILURE >> > Recording test results >> > Email was triggered for: Failure - Any >> > Sending email for trigger: Failure - Any >> > >> > >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
