Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/855/
2 tests failed. REGRESSION: org.apache.solr.cloud.RecoveryZkTest.test Error Message: shard1 is not consistent. Got 864 from http://127.0.0.1:61546/up_lvc/gp/collection1lastClient and got 223 from http://127.0.0.1:61558/up_lvc/gp/collection1 Stack Trace: java.lang.AssertionError: shard1 is not consistent. Got 864 from http://127.0.0.1:61546/up_lvc/gp/collection1lastClient and got 223 from http://127.0.0.1:61558/up_lvc/gp/collection1 at __randomizedtesting.SeedInfo.seed([19F35FD276FB405C:91A76008D8072DA4]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:123) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) REGRESSION: org.apache.solr.update.processor.UpdateRequestProcessorFactoryTest.testUpdateDistribChainSkipping Error Message: Stack Trace: java.lang.AssertionError at __randomizedtesting.SeedInfo.seed([19F35FD276FB405C:6817A10400106790]:0) at org.junit.Assert.fail(Assert.java:92) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.solr.update.processor.UpdateRequestProcessorFactoryTest.testUpdateDistribChainSkipping(UpdateRequestProcessorFactoryTest.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 11091 lines...] [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest [junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/init-core-data-001 [junit4] 2> 2066405 T12843 oejs.Server.doStart jetty-9.2.10.v20150310 [junit4] 2> 2066408 T12843 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@3122ead5{/up_lvc/gp,null,AVAILABLE} [junit4] 2> 2066415 T12843 oejs.AbstractConnector.doStart Started ServerConnector@4f97eee0{HTTP/1.1}{127.0.0.1:61539} [junit4] 2> 2066415 T12843 oejs.Server.doStart Started @2071480ms [junit4] 2> 2066525 T12843 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/control-001/lib). [junit4] 2> 2068111 T12875 n:127.0.0.1:61539_up_lvc%2Fgp x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index... [junit4] 2> 2068149 T12875 n:127.0.0.1:61539_up_lvc%2Fgp x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index [junit4] 2> 2068149 T12875 n:127.0.0.1:61539_up_lvc%2Fgp x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501887262587092992 [junit4] 2> 2070715 T12843 oejs.Server.doStart jetty-9.2.10.v20150310 [junit4] 2> 2070724 T12843 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@107409ac{/up_lvc/gp,null,AVAILABLE} [junit4] 2> 2070725 T12843 oejs.AbstractConnector.doStart Started ServerConnector@1f601db5{HTTP/1.1}{127.0.0.1:61546} [junit4] 2> 2070726 T12843 oejs.Server.doStart Started @2075791ms [junit4] 2> 2070970 T12843 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/shard-1-001/lib). [junit4] 2> 2073661 T12900 n:127.0.0.1:61546_up_lvc%2Fgp x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index... [junit4] 2> 2073713 T12900 n:127.0.0.1:61546_up_lvc%2Fgp x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index [junit4] 2> 2073713 T12900 n:127.0.0.1:61546_up_lvc%2Fgp x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501887268421369856 [junit4] 2> 2075630 T12843 oejs.Server.doStart jetty-9.2.10.v20150310 [junit4] 2> 2075633 T12843 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@55bdb0ed{/up_lvc/gp,null,AVAILABLE} [junit4] 2> 2075634 T12843 oejs.AbstractConnector.doStart Started ServerConnector@730e77d9{HTTP/1.1}{127.0.0.1:61558} [junit4] 2> 2075634 T12843 oejs.Server.doStart Started @2080699ms [junit4] 2> 2075751 T12843 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/shard-2-001/lib). [junit4] 2> 2078344 T12922 n:127.0.0.1:61558_up_lvc%2Fgp x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index... [junit4] 2> 2078379 T12922 n:127.0.0.1:61558_up_lvc%2Fgp x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index [junit4] 2> 2078380 T12922 n:127.0.0.1:61558_up_lvc%2Fgp x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501887273315074048 [junit4] 2> 2080389 T12843 oasc.RecoveryStrategy.close WARN Stopping recovery for core=collection1 coreNodeName=core_node2 [junit4] 2> ASYNC NEW_CORE C2554 name=collection1 org.apache.solr.core.SolrCore@20c2e47d url=http://127.0.0.1:61558/up_lvc/gp/collection1 node=127.0.0.1:61558_up_lvc%2Fgp C2554_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:61558_up_lvc%2Fgp, core=collection1, base_url=http://127.0.0.1:61558/up_lvc/gp, state=recovering} [junit4] 2> 2085409 T12927 n:127.0.0.1:61558_up_lvc%2Fgp c:collection1 s:shard1 x:collection1 C2554 oasu.PeerSync.sync WARN no frame of reference to tell if we've missed updates [junit4] 2> 2085423 T12937 n:127.0.0.1:61546_up_lvc%2Fgp c:collection1 s:shard1 r:core_node1 x:collection1 oascsi.ConcurrentUpdateSolrClient$Runner.run WARN Failed to parse error response from http://127.0.0.1:61558/up_lvc/gp/collection1 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format [junit4] 2> 2085423 T12937 n:127.0.0.1:61546_up_lvc%2Fgp c:collection1 s:shard1 r:core_node1 x:collection1 oasu.StreamingSolrClients$1.handleError ERROR error org.apache.solr.common.SolrException: Can not find: /up_lvc/gp/collection1/update [junit4] 2> [junit4] 2> [junit4] 2> [junit4] 2> request: http://127.0.0.1:61558/up_lvc/gp/collection1/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A61546%2Fup_lvc%2Fgp%2Fcollection1%2F&wt=javabin&version=2 [junit4] 2> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> ASYNC NEW_CORE C2555 name=collection1 org.apache.solr.core.SolrCore@551dc08d url=http://127.0.0.1:61546/up_lvc/gp/collection1 node=127.0.0.1:61546_up_lvc%2Fgp C2555_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:61546_up_lvc%2Fgp, core=collection1, base_url=http://127.0.0.1:61546/up_lvc/gp, state=active, leader=true} [junit4] 2> 2085427 T12890 n:127.0.0.1:61546_up_lvc%2Fgp c:collection1 s:shard1 r:core_node1 x:collection1 C2555 oasup.DistributedUpdateProcessor.doFinish WARN Error sending update to http://127.0.0.1:61558/up_lvc/gp org.apache.solr.common.SolrException: Can not find: /up_lvc/gp/collection1/update [junit4] 2> [junit4] 2> [junit4] 2> [junit4] 2> request: http://127.0.0.1:61558/up_lvc/gp/collection1/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A61546%2Fup_lvc%2Fgp%2Fcollection1%2F&wt=javabin&version=2 [junit4] 2> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 2085473 T12921 n:127.0.0.1:61558_up_lvc%2Fgp oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 2085487 T12843 oejs.AbstractConnector.doStop Stopped ServerConnector@730e77d9{HTTP/1.1}{127.0.0.1:0} [junit4] 2> 2085488 T12843 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@55bdb0ed{/up_lvc/gp,null,UNAVAILABLE} [junit4] 2> 2085525 T12890 n:127.0.0.1:61546_up_lvc%2Fgp c:collection1 s:shard1 r:core_node1 x:collection1 C2555 oasc.ZkController.ensureReplicaInLeaderInitiatedRecovery WARN Leader is publishing core=collection1 coreNodeName =core_node2 state=down on behalf of un-reachable replica http://127.0.0.1:61558/up_lvc/gp/collection1/; forcePublishState? false [junit4] 2> 2085526 T12890 n:127.0.0.1:61546_up_lvc%2Fgp c:collection1 s:shard1 r:core_node1 x:collection1 C2555 oasup.DistributedUpdateProcessor.doFinish ERROR Setting up to try to start recovery on replica http://127.0.0.1:61558/up_lvc/gp/collection1/ org.apache.solr.common.SolrException: Can not find: /up_lvc/gp/collection1/update [junit4] 2> [junit4] 2> [junit4] 2> [junit4] 2> request: http://127.0.0.1:61558/up_lvc/gp/collection1/update?update.distrib=FROMLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A61546%2Fup_lvc%2Fgp%2Fcollection1%2F&wt=javabin&version=2 [junit4] 2> at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:251) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 2085690 T12843 oejs.Server.doStart jetty-9.2.10.v20150310 [junit4] 2> 2085693 T12843 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5958d672{/up_lvc/gp,null,AVAILABLE} [junit4] 2> 2085694 T12843 oejs.AbstractConnector.doStart Started ServerConnector@2c4459cb{HTTP/1.1}{127.0.0.1:61558} [junit4] 2> 2085694 T12843 oejs.Server.doStart Started @2090759ms [junit4] 2> 2085822 T12843 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/shard-2-001/lib). [junit4] 2> 2087528 T12967 n:127.0.0.1:61558_up_lvc%2Fgp x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index... [junit4] 2> 2087579 T12967 n:127.0.0.1:61558_up_lvc%2Fgp x:collection1 oasu.VersionInfo.getMaxVersionFromIndex WARN No terms found for _version_, cannot seed version bucket highest value from index [junit4] 2> 2087579 T12967 n:127.0.0.1:61558_up_lvc%2Fgp x:collection1 oasu.UpdateLog.seedBucketsWithHighestVersion WARN Could not find max version in index or recent updates, using new clock 1501887282960924672 [junit4] 2> 2088540 T12981 n:127.0.0.1:61558_up_lvc%2Fgp oasc.RecoveryStrategy.close WARN Stopping recovery for core=collection1 coreNodeName=core_node2 [junit4] 2> ASYNC NEW_CORE C2556 name=collection1 org.apache.solr.core.SolrCore@1e1df496 url=http://127.0.0.1:61558/up_lvc/gp/collection1 node=127.0.0.1:61558_up_lvc%2Fgp C2556_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:61558_up_lvc%2Fgp, core=collection1, base_url=http://127.0.0.1:61558/up_lvc/gp, state=recovering} [junit4] 2> 2088543 T12972 n:127.0.0.1:61558_up_lvc%2Fgp c:collection1 s:shard1 x:collection1 C2556 oasc.SolrException.log ERROR Error while trying to recover. core=collection1:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:61546/up_lvc/gp [junit4] 2> at java.util.concurrent.FutureTask.report(FutureTask.java:122) [junit4] 2> at java.util.concurrent.FutureTask.get(FutureTask.java:188) [junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:592) [junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:363) [junit4] 2> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:229) [junit4] 2> Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:61546/up_lvc/gp [junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574) [junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:271) [junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:267) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> Caused by: java.net.SocketException: Socket closed [junit4] 2> at java.net.SocketInputStream.socketRead0(Native Method) [junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:152) [junit4] 2> at java.net.SocketInputStream.read(SocketInputStream.java:122) [junit4] 2> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) [junit4] 2> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) [junit4] 2> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) [junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) [junit4] 2> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) [junit4] 2> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) [junit4] 2> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) [junit4] 2> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) [junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) [junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) [junit4] 2> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) [junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) [junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) [junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) [junit4] 2> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) [junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466) [junit4] 2> ... 7 more [junit4] 2> [junit4] 2> added docs:857 with 0 fails deletes:428 [junit4] 2> added docs:872 with 0 fails deletes:437 [junit4] 2> ######shard1 is not consistent. Got 864 from http://127.0.0.1:61546/up_lvc/gp/collection1lastClient and got 223 from http://127.0.0.1:61558/up_lvc/gp/collection1 [junit4] 2> ######http://127.0.0.1:61546/up_lvc/gp/collection1: SolrDocumentList[sz=864]=[SolrDocument{id=1-1, _version_=1501887273379037185}, SolrDocument{id=1-10, _version_=1501887273519546368}, SolrDocument{id=1-103, _version_=1501887275006427136}, SolrDocument{id=1-11, _version_=1501887273540517888}, SolrDocument{id=1-114, _version_=1501887275214045184}] , [...] , [SolrDocument{id=2-90, _version_=1501887274830266369}, SolrDocument{id=2-91, _version_=1501887274842849280}, SolrDocument{id=2-94, _version_=1501887274886889472}, SolrDocument{id=2-95, _version_=1501887274907860992}, SolrDocument{id=2-97, _version_=1501887274929881088}] [junit4] 2> ######http://127.0.0.1:61558/up_lvc/gp/collection1: SolrDocumentList[sz=223]=[SolrDocument{id=1-674, _version_=1501887283138134016}, SolrDocument{id=1-675, _version_=1501887283149668352}, SolrDocument{id=1-678, _version_=1501887283192659968}, SolrDocument{id=1-680, _version_=1501887283234603008}, SolrDocument{id=1-681, _version_=1501887283245088768}] , [...] , [SolrDocument{id=2-852, _version_=1501887286024863744}, SolrDocument{id=2-853, _version_=1501887286033252352}, SolrDocument{id=2-854, _version_=1501887286051078144}, SolrDocument{id=2-855, _version_=1501887286069952512}, SolrDocument{id=2-856, _version_=1501887286132867072}] [junit4] 2> ###### sizes=864,223 [junit4] 2> ###### Only in http://127.0.0.1:61546/up_lvc/gp/collection1: [{id=1-207, _version_=1501887276709314560}, {id=2-5, _version_=1501887273426223104}, {id=2-555, _version_=1501887282101092352}, {id=2-570, _version_=1501887282255233024}, {id=2-478, _version_=1501887281200365568}, {id=1-10, _version_=1501887273519546368}, {id=1-141, _version_=1501887275618795521}, {id=2-214, _version_=1501887276916932608}, {id=1-498, _version_=1501887281275863040}, {id=1-364, _version_=1501887279310831616}, {id=2-627, _version_=1501887282834046976}, {id=2-618, _version_=1501887282754355200}, {id=2-33, _version_=1501887273913810944}, {id=2-262, _version_=1501887277709656065}, {id=2-614, _version_=1501887282708217856}, {id=1-90, _version_=1501887274830266368}, {id=1-82, _version_=1501887274700242945}, {id=1-9, _version_=1501887273495429120}, {id=2-95, _version_=1501887274907860992}, {id=2-121, _version_=1501887275314708480}, {id=2-413, _version_=1501887280130818049}, {id=1-529, _version_=1501887281614553088}, {id=1-308, _version_=1501887278415347712}, {id=1-55, _version_=1501887274239918080}, {id=2-54, _version_=1501887274230480896}, {id=1-359, _version_=1501887279218556928}, {id=2-540, _version_=1501887281941708801}, {id=1-557, _version_=1501887281921785857}, {id=1-257, _version_=1501887277598507009}, {id=2-88, _version_=1501887274797760512}, {id=2-471, _version_=1501887281115430912}, {id=1-389, _version_=1501887279740747776}, {id=2-330, _version_=1501887278790737920}, {id=1-404, _version_=1501887279944171520}, {id=1-244, _version_=1501887277384597504}, {id=2-477, _version_=1501887281187782656}, {id=1-31, _version_=1501887273882353664}, {id=2-584, _version_=1501887282402033664}, {id=1-523, _version_=1501887281551638528}, {id=2-622, _version_=1501887282793152512}, {id=1-156, _version_=1501887275843190784}, {id=2-290, _version_=1501887278179418112}, {id=2-521, _version_=1501887281674321921}, {id=1-341, _version_=1501887278929149952}, {id=1-276, _version_=1501887277904691201}, {id=2-286, _version_=1501887278115454976}, {id=2-194, _version_=1501887276538396672}, {id=2-533, _version_=1501887281876697089}, {id=1-561, _version_=1501887281959534592}, {id=1-152, _version_=1501887275775033344}, {id=1-214, _version_=1501887276888621056}, {id=1-520, _version_=1501887281518084096}, {id=2-68, _version_=1501887274444390400}, {id=2-434, _version_=1501887280460070912}, {id=1-390, _version_=1501887279750184960}, {id=1-566, _version_=1501887282020352000}, {id=1-24, _version_=1501887273740795904}, {id=1-274, _version_=1501887277872185344}, {id=1-94, _version_=1501887274885840896}, {id=1-547, _version_=1501887281823219712}, {id=1-439, _version_=1501887280468459520}, {id=2-653, _version_=1501887283069976576}, {id=1-114, _version_=1501887275214045184}, {id=1-549, _version_=1501887281843142656}, {id=1-25, _version_=1501887273764913152}, {id=2-364, _version_=1501887279351726080}, {id=1-126, _version_=1501887275396497408}, {id=2-485, _version_=1501887281265377280}, {id=1-119, _version_=1501887275293736960}, {id=1-199, _version_=1501887276565659648}, {id=1-326, _version_=1501887278691123200}, {id=1-488, _version_=1501887281155276800}, {id=1-74, _version_=1501887274540859392}, {id=1-1, _version_=1501887273379037185}, {id=2-461, _version_=1501887280934027264}, {id=1-638, _version_=1501887282753306624}, {id=1-550, _version_=1501887281858871296}, {id=2-256, _version_=1501887277608992768}, {id=2-559, _version_=1501887282146181120}, {id=1-385, _version_=1501887279696707584}, {id=2-359, _version_=1501887279253159936}, {id=1-657, _version_=1501887282924224512}, {id=1-338, _version_=1501887278889304064}, {id=2-271, _version_=1501887277875331072}, {id=2-115, _version_=1501887275223482368}, {id=1-471, _version_=1501887280916201472}, {id=2-155, _version_=1501887275868356608}, {id=2-523, _version_=1501887281761353728}, {id=2-548, _version_=1501887282042372097}, {id=1-634, _version_=1501887282717655040}, {id=2-189, _version_=1501887276432490496}, {id=1-221, _version_=1501887277019693056}, {id=2-580, _version_=1501887282366382080}, {id=2-57, _version_=1501887274277666816}, {id=2-32, _version_=1501887273903325184}, {id=1-425, _version_=1501887280263987200}, {id=2-569, _version_=1501887282248941568}, {id=2-421, _version_=1501887280261890048}, {id=2-124, _version_=1501887275365040128}, {id=1-176, _version_=1501887276154617857}, {id=2-127, _version_=1501887275414323200}, {id=1-380, _version_=1501887279562489856}, {id=1-28, _version_=1501887273820487680}, {id=1-604, _version_=1501887282405179392}, {id=1-582, _version_=1501887282177638400}, {id=2-619, _version_=1501887282762743808}, {id=1-454, _version_=1501887280696000512}, {id=1-450, _version_=1501887280638328832}, {id=2-332, _version_=1501887278845263872}, {id=1-60, _version_=1501887274321707008}, {id=2-232, _version_=1501887277230456833}, {id=2-261, _version_=1501887277699170304}, {id=1-26, _version_=1501887273775398912}, {id=1-472, _version_=1501887280929832960}, {id=1-301, _version_=1501887278310490112}, {id=1-165, _version_=1501887275975311360}, {id=2-307, _version_=1501887278436319232}, {id=2-412, _version_=1501887280120332289}, {id=2-615, _version_=1501887282722897920}, {id=2-547, _version_=1501887282028740608}, {id=2-172, _version_=1501887276143083520}, {id=2-556, _version_=1501887282113675264}, {id=1-669, _version_=1501887283044810752}, {id=2-636, _version_=1501887282916884480}, {id=1-461, _version_=1501887280789323776}, {id=1-411, _version_=1501887280047980544}, {id=2-133, _version_=1501887275511840768}, {id=1-324, _version_=1501887278644985856}, {id=2-394, _version_=1501887279794225152}, {id=1-355, _version_=1501887279159836672}, {id=2-337, _version_=1501887278916567040}, {id=2-565, _version_=1501887282211192832}, {id=2-274, _version_=1501887277916225536}, {id=1-169, _version_=1501887276028788736}, {id=2-385, _version_=1501887279672590336}, {id=2-202, _version_=1501887276662128640}, {id=2-577, _version_=1501887282331779072}, {id=1-330, _version_=1501887278764523520}, {id=1-504, _version_=1501887281339826176}, {id=1-325, _version_=1501887278669103104}, {id=2-134, _version_=1501887275532812288}, {id=1-591, _version_=1501887282267815937}, {id=1-218, _version_=1501887276963069952}, {id=2-505, _version_=1501887281501306880}, {id=2-574, _version_=1501887282300321793}, {id=1-248, _version_=1501887277444366336}, {id=2-160, _version_=1501887275927076864}, {id=1-485, _version_=1501887281120673792}, {id=1-49, _version_=1501887274152886272}, {id=2-592, _version_=1501887282474385408}, {id=1-30, _version_=1501887273870819328}, {id=1-91, _version_=1501887274841800704}, {id=2-528, _version_=1501887281826365440}, {id=1-83, _version_=1501887274711777281}, {id=1-407, _version_=1501887279993454592}, {id=1-398, _version_=1501887279866576896}, {id=1-273, _version_=1501887277850165248}, {id=1-656, _version_=1501887282907447296}, {id=2-86, _version_=1501887274764206080}, {id=1-467, _version_=1501887280858529792}, {id=1-193, _version_=1501887276469190656}, {id=1-574, _version_=1501887282101092353}, {id=1-261, _version_=1501887277666664448}, {id=1-36, _version_=1501887273963094016}, {id=1-332, _version_=1501887278805417984}, {id=1-124, _version_=1501887275368185856}, {id=2-119, _version_=1501887275292688384}, {id=2-497, _version_=1501887281401692160}, {id=1-194, _version_=1501887276493307904}, {id=1-7, _version_=1501887273450340352}, {id=2-268, _version_=1501887277812416512}, {id=2-484, _version_=1501887281259085824}, {id=1-212, _version_=1501887276843532288}, {id=1-473, _version_=1501887280938221568}, {id=2-475, _version_=1501887281165762560}, {id=2-446, _version_=1501887280632037376}, {id=2-578, _version_=1501887282340167680}, {id=2-141, _version_=1501887275618795520}, {id=1-601, _version_=1501887282376867840}, {id=1-238, _version_=1501887277295468545}, {id=2-617, _version_=1501887282748063744}, {id=1-258, _version_=1501887277618429952}, {id=2-376, _version_=1501887279542566912}, {id=2-169, _version_=1501887276088557568}, {id=1-475, _version_=1501887280969678848}, {id=2-327, _version_=1501887278729920512}, {id=1-233, _version_=1501887277230456832}, {id=2-473, _version_=1501887281138499584}, {id=1-172, _version_=1501887276083314688}, {id=1-486, _version_=1501887281135353856}, {id=2-234, _version_=1501887277271351296}, {id=1-617, _version_=1501887282538348544}, {id=1-23, _version_=1501887273719824384}, {id=2-439, _version_=1501887280533471232}, {id=2-366, _version_=1501887279390523393}, {id=1-502, _version_=1501887281320951808}, {id=2-263, _version_=1501887277719093248}, {id=1-116, _version_=1501887275239211008}, {id=1-533, _version_=1501887281660690432}, {id=2-462, _version_=1501887280951853056}, {id=2-610, _version_=1501887282657886209}, {id=2-502, _version_=1501887281465655297}, {id=1-32, _version_=1501887273902276608}, {id=1-366, _version_=1501887279352774656}, {id=2-16, _version_=1501887273607626752}, {id=2-654, _version_=1501887283077316608}, {id=2-283, _version_=1501887278078754816}, {id=2-80, _version_=1501887274677174273}, {id=2-642, _version_=1501887282976653312}, {id=2-549, _version_=1501887282048663552}, {id=1-264, _version_=1501887277698121728}, {id=1-615, _version_=1501887282508988416}, {id=1-598, _version_=1501887282357993473}, {id=2-84, _version_=1501887274731700224}, {id=2-179, _version_=1501887276273106944}, {id=2-498, _version_=1501887281416372225}, {id=2-454, _version_=1501887280855384064}, {id=2-326, _version_=1501887278720483328}, {id=1-618, _version_=1501887282544640000}, {id=2-607, _version_=1501887282628526080}, {id=1-59, _version_=1501887274311221248}, {id=1-118, _version_=1501887275270668288}, {id=1-540, _version_=1501887281731993600}, {id=2-51, _version_=1501887274185392129}, {id=2-36, _version_=1501887273966239744}, {id=1-660, _version_=1501887282950438913}, {id=2-60, _version_=1501887274319609856}, {id=1-474, _version_=1501887280954998784}, {id=1-57, _version_=1501887274279763968}, {id=1-252, _version_=1501887277506232320}, {id=2-355, _version_=1501887279198633984}, {id=2-495, _version_=1501887281375477760}, {id=2-55, _version_=1501887274240966656}, {id=2-154, _version_=1501887275847385088}, {id=1-307, _version_=1501887278396473344}, {id=2-595, _version_=1501887282506891264}, {id=1-302, _version_=1501887278319927296}, {id=1-275, _version_=1501887277893156864}, {id=1-51, _version_=1501887274185392128}, {id=2-491, _version_=1501887281330388992}, {id=1-294, _version_=1501887278181515264}, {id=2-239, _version_=1501887277327974400}, {id=1-420, _version_=1501887280192684032}, {id=2-207, _version_=1501887276743917568}, {id=1-392, _version_=1501887279778496512}, {id=1-586, _version_=1501887282212241408}, {id=2-338, _version_=1501887278935441408}, {id=2-581, _version_=1501887282377916416}, {id=2-117, _version_=1501887275257036800}, {id=2-551, _version_=1501887282068586496}, {id=1-327, _version_=1501887278700560384}, {id=1-267, _version_=1501887277759987712}, {id=2-407, _version_=1501887279984017408}, {id=1-424, _version_=1501887280247209984}, {id=1-29, _version_=1501887273845653504}, {id=2-193, _version_=1501887276516376576}, {id=2-254, _version_=1501887277576486913}, {id=1-254, _version_=1501887277547126784}, {id=2-91, _version_=1501887274842849280}, {id=1-469, _version_=1501887280886841344}, {id=1-607, _version_=1501887282430345216}, {id=2-469, _version_=1501887281042030592}, {id=1-421, _version_=1501887280202121216}, {id=2-257, _version_=1501887277631012864}, {id=1-406, _version_=1501887279975628800}, {id=1-318, _version_=1501887278553759744}, {id=1-39, _version_=1501887274006085632}, {id=2-507, _version_=1501887281520181248}, {id=2-499, _version_=1501887281432100864}, {id=1-224, _version_=1501887277052198912}, {id=2-362, _version_=1501887279310831617}, {id=1-319, _version_=1501887278571585536}, {id=2-392, _version_=1501887279757524992}, {id=2-591, _version_=1501887282467045377}, {id=2-223, _version_=1501887277066878976}, {id=2-415, _version_=1501887280167518208}, {id=1-551, _version_=1501887281865162752}, {id=1-262, _version_=1501887277676101632}, {id=1-417, _version_=1501887280130818048}, {id=2-25, _version_=1501887273763864576}, {id=2-638, _version_=1501887282936807425}, {id=2-472, _version_=1501887281131159552}, {id=1-271, _version_=1501887277830242304}, {id=1-668, _version_=1501887283029082113}, {id=1-624, _version_=1501887282608603136}, {id=2-424, _version_=1501887280311173120}, {id=1-493, _version_=1501887281219239936}, {id=2-626, _version_=1501887282827755520}, {id=2-393, _version_=1501887279776399360}, {id=2-427, _version_=1501887280365699072}, {id=2-313, _version_=1501887278505525248}, {id=2-319, _version_=1501887278608285696}, {id=2-294, _version_=1501887278242332673}, {id=2-192, _version_=1501887276494356480}, {id=2-537, _version_=1501887281902911488}, {id=2-81, _version_=1501887274688708608}, {id=1-11, _version_=1501887273540517888}, {id=2-11, _version_=1501887273540517889}, {id=2-325, _version_=1501887278699511808}, {id=2-211, _version_=1501887276862406656}, {id=1-42, _version_=1501887274040688640}, {id=1-376, _version_=1501887279504818177}, {id=2-204, _version_=1501887276686245888}, {id=1-219, _version_=1501887276974604288}, {id=1-35, _version_=1501887273943171072}, {id=1-590, _version_=1501887282261524481}, {id=2-467, _version_=1501887281011621888}, {id=1-337, _version_=1501887278872526848}, {id=1-534, _version_=1501887281666981888}, {id=1-293, _version_=1501887278174175232}, {id=2-346, _version_=1501887279044493312}, {id=1-478, _version_=1501887281005330432}, {id=2-184, _version_=1501887276352798720}, {id=1-127, _version_=1501887275415371776}, {id=2-402, _version_=1501887279905374209}, {id=2-259, _version_=1501887277668761600}, {id=1-346, _version_=1501887278997307392}, {id=1-565, _version_=1501887282007769088}, {id=2-443, _version_=1501887280582754304}, {id=1-477, _version_=1501887280991698944}, {id=1-521, _version_=1501887281531715584}, {id=2-305, _version_=1501887278408007680}, {id=2-590, _version_=1501887282460753920}, {id=1-559, _version_=1501887281941708800}, {id=1-201, _version_=1501887276604456960}, {id=1-195, _version_=1501887276504842241}, {id=1-370, _version_=1501887279403106304}, {id=2-539, _version_=1501887281928077312}, {id=1-408, _version_=1501887280011280384}, {id=1-548, _version_=1501887281835802624}, {id=2-289, _version_=1501887278159495168}, {id=1-499, _version_=1501887281289494528}, {id=2-323, _version_=1501887278669103105}, {id=1-204, _version_=1501887276640108544}, {id=1-654, _version_=1501887282893815808}, {id=2-531, _version_=1501887281855725568}, {id=2-520, _version_=1501887281658593280}, {id=2-135, _version_=1501887275543298048}, {id=1-121, _version_=1501887275316805632}, {id=1-47, _version_=1501887274118283264}, {id=2-182, _version_=1501887276318195712}, {id=2-292, _version_=1501887278221361152}, {id=2-380, _version_=1501887279599190016}, {id=1-556, _version_=1501887281909202945}, {id=1-245, _version_=1501887277404520448}, {id=1-84, _version_=1501887274732748800}, {id=1-103, _version_=1501887275006427136}, {id=2-53, _version_=1501887274217897985}, {id=2-219, _version_=1501887277001867264}, {id=1-443, _version_=1501887280525082624}, {id=2-216, _version_=1501887276956778496}, {id=2-557, _version_=1501887282126258176}, {id=2-534, _version_=1501887281884037120}, {id=2-22, _version_=1501887273710387200}, {id=2-186, _version_=1501887276397887488}, {id=1-189, _version_=1501887276387401729}, {id=1-53, _version_=1501887274217897984}, {id=2-83, _version_=1501887274711777280}, {id=1-514, _version_=1501887281458315264}, {id=1-648, _version_=1501887282845581312}, {id=2-245, _version_=1501887277426540544}, {id=1-335, _version_=1501887278845263873}, {id=2-561, _version_=1501887282167152640}, {id=2-23, _version_=1501887273721921536}, {id=2-488, _version_=1501887281296834560}, {id=1-208, _version_=1501887276721897472}, {id=1-237, _version_=1501887277281837056}, {id=2-586, _version_=1501887282415665152}, {id=2-331, _version_=1501887278826389504}, {id=2-425, _version_=1501887280332144640}, {id=2-235, _version_=1501887277282885632}, {id=1-430, _version_=1501887280339484672}, {id=1-289, _version_=1501887278112309248}, {id=1-560, _version_=1501887281948000256}, {id=1-528, _version_=1501887281601970176}, {id=2-389, _version_=1501887279718727680}, {id=2-39, _version_=1501887274010279936}, {id=1-215, _version_=1501887276911689728}, {id=2-156, _version_=1501887275881988096}, {id=1-614, _version_=1501887282495356928}, {id=2-308, _version_=1501887278455193600}, { [junit4] 2> id=1-22, _version_=1501887273708290048}, {id=2-176, _version_=1501887276210192384}, {id=2-293, _version_=1501887278230798336}, {id=1-67, _version_=1501887274431807488}, {id=1-192, _version_=1501887276446121984}, {id=1-292, _version_=1501887278165786624}, {id=2-288, _version_=1501887278136426496}, {id=2-624, _version_=1501887282813075456}, {id=1-177, _version_=1501887276165103617}, {id=2-273, _version_=1501887277895254016}, {id=1-180, _version_=1501887276214386688}, {id=2-601, _version_=1501887282553028608}, {id=1-531, _version_=1501887281640767488}, {id=2-634, _version_=1501887282904301568}, {id=1-595, _version_=1501887282306613248}, {id=2-287, _version_=1501887278126989312}, {id=1-234, _version_=1501887277251428352}, {id=2-28, _version_=1501887273820487681}, {id=2-165, _version_=1501887276016205824}, {id=1-95, _version_=1501887274906812416}, {id=2-30, _version_=1501887273871867904}, {id=1-286, _version_=1501887278076657664}, {id=2-648, _version_=1501887283029082112}, {id=2-545, _version_=1501887282001477632}, {id=1-155, _version_=1501887275832705024}, {id=1-288, _version_=1501887278094483456}, {id=1-619, _version_=1501887282550931456}, {id=2-567, _version_=1501887282230067200}, {id=1-464, _version_=1501887280824975360}, {id=2-212, _version_=1501887276873940992}, {id=1-592, _version_=1501887282275155968}, {id=2-315, _version_=1501887278533836800}, {id=1-645, _version_=1501887282820415488}, {id=2-406, _version_=1501887279964094465}, {id=1-361, _version_=1501887279254208512}, {id=1-290, _version_=1501887278131183616}, {id=1-211, _version_=1501887276830949376}, {id=1-81, _version_=1501887274688708609}, {id=1-140, _version_=1501887275606212608}, {id=1-255, _version_=1501887277557612544}, {id=2-226, _version_=1501887277110919168}, {id=1-597, _version_=1501887282351702016}, {id=1-658, _version_=1501887282936807424}, {id=2-616, _version_=1501887282735480832}, {id=1-154, _version_=1501887275810684928}, {id=2-255, _version_=1501887277588021249}, {id=1-622, _version_=1501887282578194432}, {id=2-302, _version_=1501887278368161792}, {id=1-569, _version_=1501887282048663553}, {id=2-74, _version_=1501887274537713664}, {id=2-335, _version_=1501887278885109760}, {id=2-126, _version_=1501887275395448832}, {id=2-517, _version_=1501887281631330304}, {id=2-196, _version_=1501887276566708224}, {id=1-80, _version_=1501887274677174272}, {id=2-152, _version_=1501887275813830656}, {id=2-31, _version_=1501887273883402240}, {id=1-182, _version_=1501887276276252672}, {id=2-244, _version_=1501887277408714752}, {id=1-240, _version_=1501887277325877248}, {id=2-128, _version_=1501887275425857537}, {id=1-427, _version_=1501887280292298752}, {id=1-223, _version_=1501887277040664576}, {id=2-264, _version_=1501887277729579008}, {id=2-26, _version_=1501887273775398913}, {id=1-259, _version_=1501887277638352896}, {id=1-394, _version_=1501887279817293824}, {id=1-287, _version_=1501887278086094848}, {id=1-666, _version_=1501887283009159169}, {id=2-411, _version_=1501887280102506496}, {id=2-404, _version_=1501887279931588608}, {id=1-537, _version_=1501887281689001984}, {id=2-94, _version_=1501887274886889472}, {id=1-589, _version_=1501887282253135872}, {id=1-577, _version_=1501887282127306752}, {id=2-474, _version_=1501887281151082496}, {id=2-275, _version_=1501887277934051328}, {id=1-174, _version_=1501887276118966272}, {id=1-239, _version_=1501887277305954305}, {id=1-159, _version_=1501887275877793792}, {id=2-114, _version_=1501887275210899456}, {id=2-258, _version_=1501887277648838656}, {id=1-331, _version_=1501887278785495040}, {id=2-158, _version_=1501887275904008192}, {id=1-539, _version_=1501887281719410688}, {id=2-103, _version_=1501887275005378560}, {id=2-64, _version_=1501887274386718720}, {id=1-336, _version_=1501887278854701056}, {id=1-16, _version_=1501887273608675328}, {id=2-450, _version_=1501887280694951936}, {id=2-408, _version_=1501887280002891777}, {id=2-233, _version_=1501887277251428353}, {id=2-582, _version_=1501887282383159296}, {id=2-417, _version_=1501887280195829760}, {id=1-633, _version_=1501887282709266432}, {id=2-1, _version_=1501887273379037184}, {id=1-160, _version_=1501887275890376704}, {id=2-97, _version_=1501887274929881088}, {id=2-390, _version_=1501887279728164864}, {id=2-598, _version_=1501887282532057089}, {id=2-606, _version_=1501887282615943168}, {id=1-134, _version_=1501887275533860864}, {id=1-584, _version_=1501887282197561344}, {id=1-555, _version_=1501887281897668608}, {id=2-221, _version_=1501887277045907456}, {id=1-495, _version_=1501887281238114304}, {id=1-88, _version_=1501887274796711936}, {id=2-67, _version_=1501887274433904640}, {id=1-627, _version_=1501887282630623232}, {id=2-76, _version_=1501887274582802432}, {id=2-90, _version_=1501887274830266369}, {id=1-162, _version_=1501887275921833984}, {id=2-159, _version_=1501887275915542528}, {id=2-79, _version_=1501887274638376960}, {id=2-414, _version_=1501887280157032448}, {id=2-265, _version_=1501887277750550529}, {id=1-412, _version_=1501887280067903488}, {id=1-581, _version_=1501887282169249792}, {id=1-117, _version_=1501887275259133952}, {id=1-133, _version_=1501887275511840769}, {id=2-59, _version_=1501887274309124096}, {id=2-468, _version_=1501887281025253376}, {id=2-420, _version_=1501887280252452864}, {id=2-297, _version_=1501887278284275712}, {id=1-545, _version_=1501887281793859584}, {id=2-195, _version_=1501887276553076736}, {id=1-484, _version_=1501887281069293568}, {id=2-42, _version_=1501887274043834368}, {id=1-564, _version_=1501887282000429056}, {id=1-315, _version_=1501887278503428096}, {id=2-637, _version_=1501887282929467392}, {id=2-224, _version_=1501887277077364736}, {id=2-7, _version_=1501887273448243200}, {id=2-218, _version_=1501887276990332928}, {id=1-637, _version_=1501887282747015168}, {id=1-235, _version_=1501887277261914112}, {id=2-430, _version_=1501887280410787840}, {id=1-580, _version_=1501887282153521152}, {id=2-82, _version_=1501887274700242944}, {id=2-589, _version_=1501887282453413889}, {id=2-318, _version_=1501887278587314176}, {id=1-268, _version_=1501887277779910656}, {id=2-208, _version_=1501887276755451904}, {id=1-179, _version_=1501887276202852352}, {id=1-610, _version_=1501887282453413888}, {id=2-162, _version_=1501887275959582720}, {id=1-414, _version_=1501887280094117888}, {id=1-505, _version_=1501887281352409088}, {id=2-118, _version_=1501887275270668289}, {id=1-334, _version_=1501887278836875264}, {id=1-313, _version_=1501887278473019392}, {id=1-115, _version_=1501887275227676672}, {id=1-653, _version_=1501887282887524352}, {id=2-464, _version_=1501887280975970304}, {id=1-76, _version_=1501887274582802433}, {id=2-324, _version_=1501887278679588865}, {id=2-566, _version_=1501887282223775744}, {id=2-514, _version_=1501887281596727296}, {id=1-79, _version_=1501887274638376961}, {id=2-442, _version_=1501887280573317120}, {id=2-604, _version_=1501887282598117376}, {id=1-626, _version_=1501887282623283200}, {id=1-5, _version_=1501887273426223105}, {id=2-237, _version_=1501887277305954304}, {id=2-248, _version_=1501887277469532160}, {id=1-507, _version_=1501887281371283456}, {id=1-578, _version_=1501887282133598208}, {id=1-606, _version_=1501887282417762304}, {id=2-504, _version_=1501887281488723968}, {id=2-276, _version_=1501887277944537088}, {id=1-64, _version_=1501887274386718721}, {id=2-177, _version_=1501887276221726720}, {id=2-633, _version_=1501887282898010112}, {id=2-238, _version_=1501887277316440065}, {id=1-446, _version_=1501887280577511424}, {id=1-68, _version_=1501887274443341824}, {id=1-297, _version_=1501887278231846912}, {id=2-281, _version_=1501887278047297536}, {id=2-199, _version_=1501887276614942720}, {id=1-232, _version_=1501887277206339584}, {id=1-97, _version_=1501887274930929664}, {id=2-370, _version_=1501887279440855040}, {id=2-301, _version_=1501887278358724608}, {id=1-434, _version_=1501887280395059200}, {id=1-128, _version_=1501887275425857536}, {id=1-158, _version_=1501887275866259456}, {id=1-202, _version_=1501887276615991296}, {id=1-616, _version_=1501887282523668480}, {id=2-35, _version_=1501887273946316800}, {id=2-334, _version_=1501887278874624000}, {id=1-362, _version_=1501887279272034305}, {id=1-226, _version_=1501887277083656192}, {id=1-323, _version_=1501887278634500096}, {id=1-186, _version_=1501887276354895872}, {id=1-663, _version_=1501887282990284800}, {id=1-184, _version_=1501887276310855680}, {id=1-600, _version_=1501887282370576384}, {id=2-529, _version_=1501887281838948352}, {id=2-174, _version_=1501887276174540800}, {id=2-295, _version_=1501887278253867008}, {id=2-493, _version_=1501887281356603392}, {id=2-215, _version_=1501887276944195584}, {id=1-295, _version_=1501887278190952448}, {id=1-33, _version_=1501887273912762368}, {id=2-597, _version_=1501887282525765632}, {id=1-567, _version_=1501887282026643456}, {id=1-402, _version_=1501887279915859968}, {id=1-216, _version_=1501887276923224064}, {id=1-283, _version_=1501887278034714624}, {id=1-263, _version_=1501887277686587392}, {id=1-642, _version_=1501887282795249664}, {id=1-305, _version_=1501887278369210368}, {id=1-636, _version_=1501887282731286528}, {id=2-336, _version_=1501887278894546944}, {id=1-371, _version_=1501887279421980673}, {id=2-180, _version_=1501887276284641280}, {id=2-183, _version_=1501887276339167232}, {id=1-86, _version_=1501887274765254656}, {id=1-265, _version_=1501887277720141824}, {id=2-116, _version_=1501887275235016704}, {id=1-281, _version_=1501887278003257344}, {id=1-497, _version_=1501887281263280128}, {id=2-47, _version_=1501887274115137536}, {id=2-361, _version_=1501887279293005824}, {id=2-645, _version_=1501887283002867712}, {id=2-398, _version_=1501887279842459648}, {id=2-267, _version_=1501887277791444992}, {id=2-600, _version_=1501887282546737152}, {id=1-442, _version_=1501887280514596864}, {id=1-415, _version_=1501887280103555072}, {id=2-564, _version_=1501887282204901376}, {id=1-54, _version_=1501887274228383744}, {id=2-24, _version_=1501887273741844480}, {id=2-550, _version_=1501887282061246465}, {id=1-309, _version_=1501887278432124928}, {id=1-517, _version_=1501887281492918272}, {id=1-183, _version_=1501887276299321344}, {id=2-309, _version_=1501887278465679360}, {id=1-393, _version_=1501887279797370880}, {id=1-196, _version_=1501887276516376577}, {id=2-49, _version_=1501887274149740544}, {id=1-659, _version_=1501887282943098880}, {id=1-570, _version_=1501887282054955009}, {id=2-29, _version_=1501887273845653505}, {id=2-140, _version_=1501887275607261184}, {id=2-486, _version_=1501887281277960192}, {id=1-135, _version_=1501887275543298049}, {id=2-560, _version_=1501887282152472576}, {id=1-468, _version_=1501887280873209856}, {id=1-491, _version_=1501887281191976960}, {id=2-341, _version_=1501887278975287296}, {id=2-252, _version_=1501887277535592448}, {id=2-201, _version_=1501887276649545728}, {id=2-240, _version_=1501887277349994496}, {id=2-371, _version_=1501887279459729408}, {id=2-9, _version_=1501887273496477696}, {id=1-413, _version_=1501887280077340672}, {id=1-256, _version_=1501887277576486912}, {id=2-10, _version_=1501887273519546369}, {id=1-462, _version_=1501887280802955264}] [junit4] 2> 2233110 T13006 n:127.0.0.1:61539_up_lvc%2Fgp oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 2233112 T12843 oejs.AbstractConnector.doStop Stopped ServerConnector@4f97eee0{HTTP/1.1}{127.0.0.1:0} [junit4] 2> 2233113 T12843 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@3122ead5{/up_lvc/gp,null,UNAVAILABLE} [junit4] 2> 2233130 T13019 n:127.0.0.1:61558_up_lvc%2Fgp oasc.RecoveryStrategy.close WARN Stopping recovery for core=collection1 coreNodeName=core_node2 [junit4] 2> 2234625 T13007 n:127.0.0.1:61546_up_lvc%2Fgp oascc.ZkStateReader.getIndividualColls WARN Error reading collections nodes org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/state.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getIndividualColls(ZkStateReader.java:512) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.constructState(ZkStateReader.java:467) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.access$200(ZkStateReader.java:59) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:330) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 2234626 T13007 n:127.0.0.1:61546_up_lvc%2Fgp oascc.ZkStateReader.getIndividualColls WARN Error reading collections nodes org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/control_collection/state.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getIndividualColls(ZkStateReader.java:512) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.constructState(ZkStateReader.java:467) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.access$200(ZkStateReader.java:59) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:330) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 2234627 T13010 n:127.0.0.1:61546_up_lvc%2Fgp oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 2234629 T12843 oejs.AbstractConnector.doStop Stopped ServerConnector@1f601db5{HTTP/1.1}{127.0.0.1:0} [junit4] 2> 2234629 T12843 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@107409ac{/up_lvc/gp,null,UNAVAILABLE} [junit4] 2> 2234635 T13017 n:127.0.0.1:61558_up_lvc%2Fgp oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> ASYNC NEW_CORE C2557 name=collection1 org.apache.solr.core.SolrCore@1e1df496 url=http://127.0.0.1:61558/up_lvc/gp/collection1 node=127.0.0.1:61558_up_lvc%2Fgp C2557_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:61558_up_lvc%2Fgp, core=collection1, base_url=http://127.0.0.1:61558/up_lvc/gp, state=active} [junit4] 2> 2235631 T13019 n:127.0.0.1:61558_up_lvc%2Fgp C2557 oasc.SyncStrategy.sync WARN Closed, skipping sync up. [junit4] 2> 2235646 T12843 oejs.AbstractConnector.doStop Stopped ServerConnector@2c4459cb{HTTP/1.1}{127.0.0.1:61558} [junit4] 2> 2235646 T12843 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@5958d672{/up_lvc/gp,null,UNAVAILABLE} [junit4] 2> 2235860 T12844 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2> [junit4] 2> 5 /solr/clusterstate.json [junit4] 2> 5 /solr/aliases.json [junit4] 2> 4 /solr/configs/conf1 [junit4] 2> [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2> [junit4] 2> 2 /solr/overseer_elect/election/93867953705910278-127.0.0.1:61546_up_lvc%2Fgp-n_0000000001 [junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/93867953705910278-core_node1-n_0000000000 [junit4] 2> [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2> [junit4] 2> 4 /solr/live_nodes [junit4] 2> 3 /solr/overseer/queue [junit4] 2> 3 /solr/overseer/collection-queue-work [junit4] 2> [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=test -Dtests.seed=19F35FD276FB405C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_CO -Dtests.timezone=AET -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] FAILURE 171s J3 | RecoveryZkTest.test <<< [junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 864 from http://127.0.0.1:61546/up_lvc/gp/collection1lastClient and got 223 from http://127.0.0.1:61558/up_lvc/gp/collection1 [junit4] > at __randomizedtesting.SeedInfo.seed([19F35FD276FB405C:91A76008D8072DA4]:0) [junit4] > at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:123) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.RecoveryZkTest 19F35FD276FB405C-001 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {a_i=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), rnd_b=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=PostingsFormat(name=Asserting), _version_=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_CO, timezone=AET [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=300855592,total=525860864 [junit4] 2> NOTE: All tests run in this JVM: [TermVectorComponentDistributedTest, TestSystemIdResolver, TestLeaderElectionZkExpiry, CollectionsAPIDistributedZkTest, TestCloudManagedSchemaConcurrent, OverriddenZkACLAndCredentialsProvidersTest, DeleteLastCustomShardedReplicaTest, DocValuesMultiTest, TestDefaultSimilarityFactory, TestRangeQuery, LeaderElectionTest, CoreAdminRequestStatusTest, TestFreeTextSuggestions, TestCollationFieldDocValues, TestJsonRequest, NumericFieldsTest, TestNoOpRegenerator, TestReqParamsAPI, SampleTest, DefaultValueUpdateProcessorTest, TestUtils, TestSurroundQueryParser, TestDistribDocBasedVersion, MultiTermTest, TestJmxMonitoredMap, TestReloadAndDeleteDocs, SoftAutoCommitTest, TestNRTOpen, TestSimpleTrackingShardHandler, TestClassNameShortening, TestManagedSchema, TestFiltering, TestRawResponseWriter, ConcurrentDeleteAndCreateCollectionTest, TestSolrConfigHandler, DistributedExpandComponentTest, TestCodecSupport, AnalysisAfterCoreReloadTest, TestConfigReload, ZkStateWriterTest, ResponseHeaderTest, AddBlockUpdateTest, CachingDirectoryFactoryTest, OutputWriterTest, AsyncMigrateRouteKeyTest, CloudExitableDirectoryReaderTest, TestAuthorizationFramework, TestManagedSchemaFieldResource, BadIndexSchemaTest, TestCSVLoader, TestChildDocTransformer, TestWriterPerf, DistributedFacetPivotSmallAdvancedTest, AtomicUpdatesTest, FacetPivotSmallTest, TestIndexingPerformance, VersionInfoTest, ResponseLogComponentTest, TestValueSourceCache, ClusterStateUpdateTest, ExternalCollectionsTest, BasicDistributedZkTest, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest] [junit4] Completed [258/496] on J3 in 170.92s, 1 test, 1 failure <<< FAILURES! [...truncated 205 lines...] [junit4] Suite: org.apache.solr.update.processor.UpdateRequestProcessorFactoryTest [junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.update.processor.UpdateRequestProcessorFactoryTest 19F35FD276FB405C-001/init-core-data-001 [junit4] 2> 2473552 T14893 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true) [junit4] 2> 2473553 T14893 oas.SolrTestCaseJ4.initCore ####initCore [junit4] 2> 2474047 T14893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 2474047 T14893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 2474048 T14893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 2474048 T14893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 2474083 T14893 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/lib). [junit4] 2> 2474757 T14894 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 2474758 T14894 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 2474759 T14894 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE [junit4] 2> 2474759 T14894 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING [junit4] 2> 2474760 T14894 x:collection1 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.update.processor.UpdateRequestProcessorFactoryTest 19F35FD276FB405C-001/init-core-data-001/index' doesn't exist. Creating new index... [junit4] 2> 2475015 T14894 x:collection1 oasc.RequestHandlers.initHandlersFromConfig WARN no default request handler is registered (either '/select' or 'standard') [junit4] 2> 2475024 T14894 x:collection1 oasr.ManagedResourceStorage.newStorageIO WARN Cannot write to config directory /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf; switching to use InMemory storage instead. [junit4] 2> 2475027 T14893 oas.SolrTestCaseJ4.initCore ####initCore end [junit4] 2> 2475034 T14893 oas.SolrTestCaseJ4.setUp ###Starting testUpdateDistribChainSkipping [junit4] 2> 2475035 T14893 oas.SolrTestCaseJ4.tearDown ###Ending testUpdateDistribChainSkipping [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=UpdateRequestProcessorFactoryTest -Dtests.method=testUpdateDistribChainSkipping -Dtests.seed=19F35FD276FB405C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sl_SI -Dtests.timezone=Europe/Isle_of_Man -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] FAILURE 0.01s J3 | UpdateRequestProcessorFactoryTest.testUpdateDistribChainSkipping <<< [junit4] > Throwable #1: java.lang.AssertionError [junit4] > at __randomizedtesting.SeedInfo.seed([19F35FD276FB405C:6817A10400106790]:0) [junit4] > at org.apache.solr.update.processor.UpdateRequestProcessorFactoryTest.testUpdateDistribChainSkipping(UpdateRequestProcessorFactoryTest.java:111) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] 2> 2475047 T14893 oas.SolrTestCaseJ4.setUp ###Starting testConfiguration [junit4] 2> 2475048 T14893 oas.SolrTestCaseJ4.tearDown ###Ending testConfiguration [junit4] 2> 2475049 T14893 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.update.processor.UpdateRequestProcessorFactoryTest 19F35FD276FB405C-001 [junit4] 2> NOTE: test params are: codec=Lucene50, sim=DefaultSimilarity, locale=sl_SI, timezone=Europe/Isle_of_Man [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=228639944,total=533725184 [junit4] 2> NOTE: All tests run in this JVM: [TermVectorComponentDistributedTest, TestSystemIdResolver, TestLeaderElectionZkExpiry, CollectionsAPIDistributedZkTest, TestCloudManagedSchemaConcurrent, OverriddenZkACLAndCredentialsProvidersTest, DeleteLastCustomShardedReplicaTest, DocValuesMultiTest, TestDefaultSimilarityFactory, TestRangeQuery, LeaderElectionTest, CoreAdminRequestStatusTest, TestFreeTextSuggestions, TestCollationFieldDocValues, TestJsonRequest, NumericFieldsTest, TestNoOpRegenerator, TestReqParamsAPI, SampleTest, DefaultValueUpdateProcessorTest, TestUtils, TestSurroundQueryParser, TestDistribDocBasedVersion, MultiTermTest, TestJmxMonitoredMap, TestReloadAndDeleteDocs, SoftAutoCommitTest, TestNRTOpen, TestSimpleTrackingShardHandler, TestClassNameShortening, TestManagedSchema, TestFiltering, TestRawResponseWriter, ConcurrentDeleteAndCreateCollectionTest, TestSolrConfigHandler, DistributedExpandComponentTest, TestCodecSupport, AnalysisAfterCoreReloadTest, TestConfigReload, ZkStateWriterTest, ResponseHeaderTest, AddBlockUpdateTest, CachingDirectoryFactoryTest, OutputWriterTest, AsyncMigrateRouteKeyTest, CloudExitableDirectoryReaderTest, TestAuthorizationFramework, TestManagedSchemaFieldResource, BadIndexSchemaTest, TestCSVLoader, TestChildDocTransformer, TestWriterPerf, DistributedFacetPivotSmallAdvancedTest, AtomicUpdatesTest, FacetPivotSmallTest, TestIndexingPerformance, VersionInfoTest, ResponseLogComponentTest, TestValueSourceCache, ClusterStateUpdateTest, ExternalCollectionsTest, BasicDistributedZkTest, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, TestDistributedGrouping, TestFaceting, ZkControllerTest, TestRealTimeGet, TestStressReorder, TestJoin, TestReload, DistributedTermsComponentTest, SimpleFacetsTest, SolrCoreTest, SolrCmdDistributorTest, HighlighterTest, CurrencyFieldXmlFileTest, TestCoreDiscovery, SuggesterFSTTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, TestTrie, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, TestQueryUtils, StandardRequestHandlerTest, TestQueryTypes, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, TestArbitraryIndexDir, LoggingHandlerTest, IndexSchemaTest, UpdateRequestProcessorFactoryTest] [junit4] Completed [321/496] on J3 in 1.52s, 2 tests, 1 failure <<< FAILURES! [...truncated 546 lines...] BUILD FAILED /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:543: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:484: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:229: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:511: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1433: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:991: There were test failures: 496 suites, 1971 tests, 2 failures, 65 ignored (11 assumptions) Total time: 181 minutes 40 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #852 Archived 5 artifacts Archive block size is 32768 Received 0 blocks and 909921216 bytes Compression is 0.0% Took 2 min 35 sec Recording test results Email was triggered for: Failure Sending email for trigger: Failure
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
