Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/195/

3 tests failed.
FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes

Error Message:
Could not find collection : nodes_used_collection

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : 
nodes_used_collection
        at 
__randomizedtesting.SeedInfo.seed([F6C1D2403434BA00:9281072C8FFA3211]:0)
        at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:193)
        at 
org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:216)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCoresAreDistributedAcrossNodes(CollectionsAPIDistributedZkTest.java:397)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Expected to see collection awhollynewcollection_0 null Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/4)={
   "replicationFactor":"1",   "shards":{     "shard1":{       
"range":"80000000-d554ffff",       "state":"active",       "replicas":{}},     
"shard2":{       "range":"d5550000-2aa9ffff",       "state":"active",       
"replicas":{}},     "shard3":{       "range":"2aaa0000-7fffffff",       
"state":"active",       "replicas":{}}},   "router":{"name":"compositeId"},   
"maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected to see collection awhollynewcollection_0
null
Last available state: 
DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/4)={
  "replicationFactor":"1",
  "shards":{
    "shard1":{
      "range":"80000000-d554ffff",
      "state":"active",
      "replicas":{}},
    "shard2":{
      "range":"d5550000-2aa9ffff",
      "state":"active",
      "replicas":{}},
    "shard3":{
      "range":"2aaa0000-7fffffff",
      "state":"active",
      "replicas":{}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
        at 
__randomizedtesting.SeedInfo.seed([F6C1D2403434BA00:BEB4A6F432079595]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:236)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:496)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testSpecificConfigsets

Error Message:
KeeperErrorCode = NoNode for /collections/withconfigset2

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode 
for /collections/withconfigset2
        at 
__randomizedtesting.SeedInfo.seed([F6C1D2403434BA00:DBBF9D1AC310000C]:0)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:425)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11384 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/init-core-data-001
   [junit4]   2> 345438 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 345439 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001
   [junit4]   2> 345439 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 345447 INFO  (Thread-340) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 345447 INFO  (Thread-340) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 345547 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.c.ZkTestServer start zk server on port:47793
   [junit4]   2> 345623 INFO  (jetty-launcher-223-thread-3) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 345623 INFO  (jetty-launcher-223-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 345623 INFO  (jetty-launcher-223-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 345626 INFO  (jetty-launcher-223-thread-4) [    ] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 345633 INFO  (jetty-launcher-223-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@300ac283{/solr,null,AVAILABLE}
   [junit4]   2> 345640 INFO  (jetty-launcher-223-thread-3) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@663b3020{/solr,null,AVAILABLE}
   [junit4]   2> 345643 INFO  (jetty-launcher-223-thread-3) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@13af7c5e{SSL,[ssl, 
http/1.1]}{127.0.0.1:48954}
   [junit4]   2> 345644 INFO  (jetty-launcher-223-thread-3) [    ] 
o.e.j.s.Server Started @350380ms
   [junit4]   2> 345644 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=48954}
   [junit4]   2> 345644 ERROR (jetty-launcher-223-thread-3) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 345659 INFO  (jetty-launcher-223-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@143ffe37{/solr,null,AVAILABLE}
   [junit4]   2> 345672 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 345672 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 345672 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 345672 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-10T16:33:56.830Z
   [junit4]   2> 345681 INFO  (jetty-launcher-223-thread-1) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@1c7e9c8b{SSL,[ssl, 
http/1.1]}{127.0.0.1:48587}
   [junit4]   2> 345681 INFO  (jetty-launcher-223-thread-1) [    ] 
o.e.j.s.Server Started @350419ms
   [junit4]   2> 345681 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=48587}
   [junit4]   2> 345682 ERROR (jetty-launcher-223-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@4b9d5386{SSL,[ssl, 
http/1.1]}{127.0.0.1:40953}
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.e.j.s.Server Started @350420ms
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=40953}
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-10T16:33:56.840Z
   [junit4]   2> 345682 ERROR (jetty-launcher-223-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 345682 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-10T16:33:56.840Z
   [junit4]   2> 345692 INFO  (jetty-launcher-223-thread-4) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@10cc5291{/solr,null,AVAILABLE}
   [junit4]   2> 345708 INFO  (jetty-launcher-223-thread-4) [    ] 
o.e.j.s.ServerConnector Started ServerConnector@211452e2{SSL,[ssl, 
http/1.1]}{127.0.0.1:60566}
   [junit4]   2> 345708 INFO  (jetty-launcher-223-thread-4) [    ] 
o.e.j.s.Server Started @350446ms
   [junit4]   2> 345708 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=60566}
   [junit4]   2> 345715 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 345722 ERROR (jetty-launcher-223-thread-4) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 345722 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 345722 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 345722 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 345722 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-11-10T16:33:56.880Z
   [junit4]   2> 345724 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 345724 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 345747 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 345747 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 345748 INFO  (jetty-launcher-223-thread-3) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47793/solr
   [junit4]   2> 345757 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 345763 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 345764 INFO  (jetty-launcher-223-thread-4) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47793/solr
   [junit4]   2> 345772 INFO  (jetty-launcher-223-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47793/solr
   [junit4]   2> 345773 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 345776 INFO  (jetty-launcher-223-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:47793/solr
   [junit4]   2> 345838 INFO  (jetty-launcher-223-thread-4) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:60566_solr
   [junit4]   2> 345851 INFO  (jetty-launcher-223-thread-3) 
[n:127.0.0.1:48954_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:48954_solr
   [junit4]   2> 345852 INFO  (jetty-launcher-223-thread-4) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.Overseer Overseer 
(id=96914350840610826-127.0.0.1:60566_solr-n_0000000000) starting
   [junit4]   2> 345858 INFO  
(zkCallback-242-thread-1-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 345858 INFO  
(zkCallback-240-thread-1-processing-n:127.0.0.1:48954_solr) 
[n:127.0.0.1:48954_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 345860 INFO  (jetty-launcher-223-thread-1) 
[n:127.0.0.1:48587_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 345875 INFO  (jetty-launcher-223-thread-2) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 345970 INFO  (jetty-launcher-223-thread-4) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:60566_solr
   [junit4]   2> 345970 INFO  (jetty-launcher-223-thread-3) 
[n:127.0.0.1:48954_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node3/.
   [junit4]   2> 345973 INFO  (jetty-launcher-223-thread-1) 
[n:127.0.0.1:48587_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:48587_solr
   [junit4]   2> 345977 INFO  
(zkCallback-243-thread-1-processing-n:127.0.0.1:40953_solr) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (3)
   [junit4]   2> 345978 INFO  
(zkCallback-244-thread-1-processing-n:127.0.0.1:48587_solr) 
[n:127.0.0.1:48587_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (3)
   [junit4]   2> 345979 INFO  
(zkCallback-242-thread-1-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (3)
   [junit4]   2> 345979 INFO  
(zkCallback-240-thread-1-processing-n:127.0.0.1:48954_solr) 
[n:127.0.0.1:48954_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (3)
   [junit4]   2> 345993 INFO  (jetty-launcher-223-thread-2) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:40953_solr
   [junit4]   2> 345994 INFO  (jetty-launcher-223-thread-4) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node4/.
   [junit4]   2> 345998 INFO  
(zkCallback-242-thread-1-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 345998 INFO  
(zkCallback-243-thread-1-processing-n:127.0.0.1:40953_solr) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 345998 INFO  
(zkCallback-240-thread-1-processing-n:127.0.0.1:48954_solr) 
[n:127.0.0.1:48954_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 345998 INFO  
(zkCallback-244-thread-1-processing-n:127.0.0.1:48587_solr) 
[n:127.0.0.1:48587_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (3) -> (4)
   [junit4]   2> 346010 INFO  (jetty-launcher-223-thread-1) 
[n:127.0.0.1:48587_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node1/.
   [junit4]   2> 346016 INFO  (jetty-launcher-223-thread-2) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node2/.
   [junit4]   2> 346060 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 347077 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your 
platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 348278 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 348704 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog
   [junit4]   2> 348740 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 348842 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 348963 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_40328_hdfs____.6425qx/webapp
   [junit4]   2> 349936 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40328
   [junit4]   2> 351166 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 351173 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 351188 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_52013_datanode____.4a7rpg/webapp
   [junit4]   2> 351695 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52013
   [junit4]   2> 352263 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 352265 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 352299 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_44939_datanode____.72620q/webapp
   [junit4]   2> 352814 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44939
   [junit4]   2> 354590 INFO  (IPC Server handler 5 on 53630) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-83749173-1659-4204-a1f8-b6154a34c35b node 
DatanodeRegistration(127.0.0.1:42118, 
datanodeUuid=10445a2a-eb7f-44dd-a193-854f45793606, infoPort=58519, 
infoSecurePort=0, ipcPort=59056, 
storageInfo=lv=-56;cid=testClusterID;nsid=804710585;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 3 msecs
   [junit4]   2> 354595 INFO  (IPC Server handler 6 on 53630) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-c6da2482-99f6-4508-8d53-ae53cbe10e92 node 
DatanodeRegistration(127.0.0.1:40732, 
datanodeUuid=6c5428c7-2126-40d8-82f5-5abb1bf54001, infoPort=46788, 
infoSecurePort=0, ipcPort=55323, 
storageInfo=lv=-56;cid=testClusterID;nsid=804710585;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 4 msecs
   [junit4]   2> 354595 INFO  (IPC Server handler 6 on 53630) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-10654d7c-36e4-4e34-9173-6487d189d1cc node 
DatanodeRegistration(127.0.0.1:40732, 
datanodeUuid=6c5428c7-2126-40d8-82f5-5abb1bf54001, infoPort=46788, 
infoSecurePort=0, ipcPort=55323, 
storageInfo=lv=-56;cid=testClusterID;nsid=804710585;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 354596 INFO  (IPC Server handler 5 on 53630) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-446fe27d-3df2-46a9-be9d-86c6dbb4bff1 node 
DatanodeRegistration(127.0.0.1:42118, 
datanodeUuid=10445a2a-eb7f-44dd-a193-854f45793606, infoPort=58519, 
infoSecurePort=0, ipcPort=59056, 
storageInfo=lv=-56;cid=testClusterID;nsid=804710585;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 1 msecs
   [junit4]   2> 354752 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[F6C1D2403434BA00])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testTooManyReplicas
   [junit4]   2> 354754 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[F6C1D2403434BA00])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 354831 INFO  (qtp516410542-1593) [n:127.0.0.1:60566_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 354837 INFO  
(OverseerThreadFactory-656-thread-1-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
collection
   [junit4]   2> 354838 WARN  
(OverseerThreadFactory-656-thread-1-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.CreateCollectionCmd Specified 
replicationFactor of 10 on collection collection is higher than or equal to the 
number of Solr instances currently live or live and part of your 
createNodeSet(4). It's unusual to run two replica of the same slice on the same 
Solr-instance.
   [junit4]   2> 354841 ERROR 
(OverseerThreadFactory-656-thread-1-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerCollectionMessageHandler 
Collection: collection operation: create 
failed:org.apache.solr.common.SolrException: Cannot create collection 
collection. Value of maxShardsPerNode is 1, and the number of nodes currently 
live or live and part of your createNodeSet is 4. This allows a maximum of 4 to 
be created. Value of numShards is 2 and value of replicationFactor is 10. This 
requires 20 shards to be created (higher than the allowed number)
   [junit4]   2>        at 
org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:148)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:224)
   [junit4]   2>        at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 354866 INFO  (qtp516410542-1593) [n:127.0.0.1:60566_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2}
 status=400 QTime=35
   [junit4]   2> 354869 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[F6C1D2403434BA00])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testTooManyReplicas
   [junit4]   2> 354965 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore-seed#[F6C1D2403434BA00])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreateShouldFailOnExistingCore
   [junit4]   2> 354967 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore-seed#[F6C1D2403434BA00])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 355043 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
dataDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-003&collection.configName=conf&name=halfcollection_shard1_replica1&action=CREATE&numShards=1&collection=halfcollectionblocker&wt=javabin&version=2
   [junit4]   2> 355046 INFO  
(OverseerStateUpdate-96914350840610826-127.0.0.1:60566_solr-n_0000000000) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 355103 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker   x:halfcollection_shard1_replica1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 355117 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker   x:halfcollection_shard1_replica1] o.a.s.s.IndexSchema 
[halfcollection_shard1_replica1] Schema name=minimal
   [junit4]   2> 355121 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker   x:halfcollection_shard1_replica1] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 355122 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker   x:halfcollection_shard1_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'halfcollection_shard1_replica1' using 
configuration from collection halfcollectionblocker
   [junit4]   2> 355143 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53630/data
   [junit4]   2> 355143 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 355144 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.SolrCore [[halfcollection_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node3/halfcollection_shard1_replica1],
 
dataDir=[hdfs://localhost:53630/data/halfcollectionblocker/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-003/]
   [junit4]   2> 355145 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:53630/data/halfcollectionblocker/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-003/snapshot_metadata
   [junit4]   2> 355172 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 355173 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 355173 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 356839 INFO  
(OverseerCollectionConfigSetProcessor-96914350840610826-127.0.0.1:60566_solr-n_0000000000)
 [n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 357238 WARN  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds 
and try again.
   [junit4]   2> 362241 WARN  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds 
and try again.
   [junit4]   2> 367537 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 367549 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:53630/data/halfcollectionblocker/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-003
   [junit4]   2> 367663 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:53630/data/halfcollectionblocker/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-003/index
   [junit4]   2> 367676 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 367676 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 367676 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 367690 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 369162 INFO  (IPC Server handler 2 on 53630) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40732 is 
added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-10654d7c-36e4-4e34-9173-6487d189d1cc:NORMAL:127.0.0.1:40732|RBW],
 
ReplicaUC[[DISK]DS-446fe27d-3df2-46a9-be9d-86c6dbb4bff1:NORMAL:127.0.0.1:42118|RBW]]}
 size 71
   [junit4]   2> 369164 INFO  (IPC Server handler 3 on 53630) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42118 is 
added to blk_1073741825_1001 size 71
   [junit4]   2> 369594 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 369594 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 369594 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 369627 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 369627 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 369763 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1334f2a7[halfcollection_shard1_replica1] main]
   [junit4]   2> 369764 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 369765 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 369768 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 369768 INFO  
(searcherExecutor-663-thread-1-processing-n:127.0.0.1:48954_solr 
x:halfcollection_shard1_replica1 s:shard1 c:halfcollectionblocker r:core_node1) 
[n:127.0.0.1:48954_solr c:halfcollectionblocker s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.SolrCore 
[halfcollection_shard1_replica1] Registered new searcher 
Searcher@1334f2a7[halfcollection_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 369768 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1550629638951141376
   [junit4]   2> 369787 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 369787 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 369787 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:48954/solr/halfcollection_shard1_replica1/
   [junit4]   2> 369787 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 369787 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.SyncStrategy 
https://127.0.0.1:48954/solr/halfcollection_shard1_replica1/ has no replicas
   [junit4]   2> 369799 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:48954/solr/halfcollection_shard1_replica1/ shard1
   [junit4]   2> 369955 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 369957 INFO  (qtp1362840958-1574) [n:127.0.0.1:48954_solr 
c:halfcollectionblocker s:shard1 r:core_node1 x:halfcollection_shard1_replica1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={dataDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-003&collection.configName=conf&name=halfcollection_shard1_replica1&action=CREATE&numShards=1&collection=halfcollectionblocker&wt=javabin&version=2}
 status=0 QTime=14914
   [junit4]   2> 370031 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
dataDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004&collection.configName=conf&name=halfcollection_shard1_replica1&action=CREATE&numShards=1&collection=halfcollectionblocker2&wt=javabin&version=2
   [junit4]   2> 370034 INFO  
(OverseerStateUpdate-96914350840610826-127.0.0.1:60566_solr-n_0000000000) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 371054 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2   x:halfcollection_shard1_replica1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 371063 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2   x:halfcollection_shard1_replica1] 
o.a.s.s.IndexSchema [halfcollection_shard1_replica1] Schema name=minimal
   [junit4]   2> 371067 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2   x:halfcollection_shard1_replica1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 371068 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2   x:halfcollection_shard1_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'halfcollection_shard1_replica1' using 
configuration from collection halfcollectionblocker2
   [junit4]   2> 371069 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:53630/data
   [junit4]   2> 371070 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos 
Authentication disabled
   [junit4]   2> 371070 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.SolrCore 
[[halfcollection_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node2/halfcollection_shard1_replica1],
 
dataDir=[hdfs://localhost:53630/data/halfcollectionblocker2/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004/]
   [junit4]   2> 371070 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:53630/data/halfcollectionblocker2/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004/snapshot_metadata
   [junit4]   2> 371086 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs 
of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 371086 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache 
target memory usage, slab size of [16777216] will allocate [1] slabs and use 
~[16777216] bytes
   [junit4]   2> 371086 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new 
single instance HDFS BlockCache
   [junit4]   2> 371104 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write 
is disabled
   [junit4]   2> 371107 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:53630/data/halfcollectionblocker2/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004
   [junit4]   2> 371152 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating 
directory factory for path 
hdfs://localhost:53630/data/halfcollectionblocker2/core_node1//x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004/index
   [junit4]   2> 371163 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs 
of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 371163 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache 
target memory usage, slab size of [16777216] will allocate [1] slabs and use 
~[16777216] bytes
   [junit4]   2> 371163 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new 
single instance HDFS BlockCache
   [junit4]   2> 371184 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write 
is disabled
   [junit4]   2> 371294 INFO  (IPC Server handler 0 on 53630) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42118 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-10654d7c-36e4-4e34-9173-6487d189d1cc:NORMAL:127.0.0.1:40732|RBW],
 
ReplicaUC[[DISK]DS-446fe27d-3df2-46a9-be9d-86c6dbb4bff1:NORMAL:127.0.0.1:42118|RBW]]}
 size 0
   [junit4]   2> 371302 INFO  (IPC Server handler 8 on 53630) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40732 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-10654d7c-36e4-4e34-9173-6487d189d1cc:NORMAL:127.0.0.1:40732|RBW],
 
ReplicaUC[[DISK]DS-446fe27d-3df2-46a9-be9d-86c6dbb4bff1:NORMAL:127.0.0.1:42118|RBW]]}
 size 0
   [junit4]   2> 371327 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog 
implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 371327 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: 
dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
numVersionBuckets=65536
   [junit4]   2> 371327 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing 
HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 371362 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: 
disabled
   [junit4]   2> 371363 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: 
disabled
   [junit4]   2> 371407 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening 
[Searcher@37d5b409[halfcollection_shard1_replica1] main]
   [junit4]   2> 371409 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured 
ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 371409 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at 
path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 371409 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be 
reserved for  10000
   [junit4]   2> 371410 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.u.UpdateLog Could not find max version 
in index or recent updates, using new clock 1550629640672903168
   [junit4]   2> 371410 INFO  
(searcherExecutor-668-thread-1-processing-n:127.0.0.1:40953_solr 
x:halfcollection_shard1_replica1 s:shard1 c:halfcollectionblocker2 
r:core_node1) [n:127.0.0.1:40953_solr c:halfcollectionblocker2 s:shard1 
r:core_node1 x:halfcollection_shard1_replica1] o.a.s.c.SolrCore 
[halfcollection_shard1_replica1] Registered new searcher 
Searcher@37d5b409[halfcollection_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 371426 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough 
replicas found to continue.
   [junit4]   2> 371427 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be 
the new leader - try and sync
   [junit4]   2> 371427 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:40953/solr/halfcollection_shard1_replica1/
   [junit4]   2> 371427 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync 
replicas to me
   [junit4]   2> 371427 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.SyncStrategy 
https://127.0.0.1:40953/solr/halfcollection_shard1_replica1/ has no replicas
   [junit4]   2> 371444 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the 
new leader: https://127.0.0.1:40953/solr/halfcollection_shard1_replica1/ shard1
   [junit4]   2> 371599 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.c.ZkController I am the leader, no 
recovery necessary
   [junit4]   2> 371603 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr 
c:halfcollectionblocker2 s:shard1 r:core_node1 
x:halfcollection_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/cores 
params={dataDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004&collection.configName=conf&name=halfcollection_shard1_replica1&action=CREATE&numShards=1&collection=halfcollectionblocker2&wt=javabin&version=2}
 status=0 QTime=1572
   [junit4]   2> 371608 INFO  (qtp516410542-1588) [n:127.0.0.1:60566_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=1&collection.configName=conf&name=halfcollection&action=CREATE&numShards=2&createNodeSet=127.0.0.1:48954_solr,127.0.0.1:40953_solr&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 371616 INFO  
(OverseerThreadFactory-656-thread-2-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
halfcollection
   [junit4]   2> 371847 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=halfcollection_shard2_replica1&action=CREATE&numShards=2&collection=halfcollection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 371869 INFO  (qtp1362840958-1573) [n:127.0.0.1:48954_solr    ] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=halfcollection_shard1_replica1&action=CREATE&numShards=2&collection=halfcollection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 371870 WARN  (qtp1362840958-1573) [n:127.0.0.1:48954_solr    ] 
o.a.s.c.CoreContainer Creating a core with existing name is not allowed
   [junit4]   2> 371870 ERROR (qtp1362840958-1573) [n:127.0.0.1:48954_solr    ] 
o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Core with name 
'halfcollection_shard1_replica1' already exists.
   [junit4]   2>        at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:764)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:156)
   [junit4]   2>        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:110)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2>        at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2>        at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2>        at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2>        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 371870 INFO  (qtp1362840958-1573) [n:127.0.0.1:48954_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=halfcollection_shard1_replica1&action=CREATE&numShards=2&collection=halfcollection&shard=shard1&wt=javabin&version=2}
 status=500 QTime=1
   [junit4]   2> 371871 ERROR (qtp1362840958-1573) [n:127.0.0.1:48954_solr    ] 
o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Core with name 
'halfcollection_shard1_replica1' already exists.
   [junit4]   2>        at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:764)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:377)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365)
   [junit4]   2>        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:156)
   [junit4]   2>        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:660)
   [junit4]   2>        at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:110)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2>        at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2>        at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2>        at 
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2>        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 371873 ERROR 
(OverseerThreadFactory-656-thread-2-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerCollectionMessageHandler Error 
from shard: https://127.0.0.1:48954/solr
   [junit4]   2> 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:48954/solr: Core with name 
'halfcollection_shard1_replica1' already exists.
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:593)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
   [junit4]   2>        at 
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]   2>        at 
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:196)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 371954 INFO  
(zkCallback-243-thread-1-processing-n:127.0.0.1:40953_solr) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/halfcollection/state.json] for collection [halfcollection] 
has occurred - updating... (live nodes size: [4])
   [junit4]   2> 372867 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2  x:halfcollection_shard2_replica1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 372877 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2  x:halfcollection_shard2_replica1] 
o.a.s.s.IndexSchema [halfcollection_shard2_replica1] Schema name=minimal
   [junit4]   2> 372880 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2  x:halfcollection_shard2_replica1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 372881 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2  x:halfcollection_shard2_replica1] 
o.a.s.c.CoreContainer Creating SolrCore 'halfcollection_shard2_replica1' using 
configuration from collection halfcollection
   [junit4]   2> 372882 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:53630/data
   [junit4]   2> 372882 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 372882 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.SolrCore [[halfcollection_shard2_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-001/node2/halfcollection_shard2_replica1],
 dataDir=[hdfs://localhost:53630/data/halfcollection/core_node1/data/]
   [junit4]   2> 372882 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:53630/data/halfcollection/core_node1/data/snapshot_metadata
   [junit4]   2> 372890 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 372890 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 372890 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 372902 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 372904 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:53630/data/halfcollection/core_node1/data
   [junit4]   2> 372934 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:53630/data/halfcollection/core_node1/data/index
   [junit4]   2> 372942 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 372942 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 372942 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 372955 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 373009 INFO  (IPC Server handler 8 on 53630) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42118 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-c6da2482-99f6-4508-8d53-ae53cbe10e92:NORMAL:127.0.0.1:40732|RBW],
 
ReplicaUC[[DISK]DS-83749173-1659-4204-a1f8-b6154a34c35b:NORMAL:127.0.0.1:42118|FINALIZED]]}
 size 0
   [junit4]   2> 373010 INFO  (IPC Server handler 9 on 53630) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40732 is 
added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-c6da2482-99f6-4508-8d53-ae53cbe10e92:NORMAL:127.0.0.1:40732|RBW],
 
ReplicaUC[[DISK]DS-83749173-1659-4204-a1f8-b6154a34c35b:NORMAL:127.0.0.1:42118|FINALIZED]]}
 size 0
   [junit4]   2> 373051 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 373051 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 373051 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 373074 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 373074 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 373110 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@51ca7b8a[halfcollection_shard2_replica1] main]
   [junit4]   2> 373113 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 373113 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 373114 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 373114 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1550629642459676672
   [junit4]   2> 373118 INFO  
(searcherExecutor-673-thread-1-processing-n:127.0.0.1:40953_solr 
x:halfcollection_shard2_replica1 s:shard2 c:halfcollection r:core_node1) 
[n:127.0.0.1:40953_solr c:halfcollection s:shard2 r:core_node1 
x:halfcollection_shard2_replica1] o.a.s.c.SolrCore 
[halfcollection_shard2_replica1] Registered new searcher 
Searcher@51ca7b8a[halfcollection_shard2_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 373129 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 373129 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 373130 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:40953/solr/halfcollection_shard2_replica1/
   [junit4]   2> 373130 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 373130 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.SyncStrategy 
https://127.0.0.1:40953/solr/halfcollection_shard2_replica1/ has no replicas
   [junit4]   2> 373136 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:40953/solr/halfcollection_shard2_replica1/ shard2
   [junit4]   2> 373241 INFO  
(zkCallback-243-thread-1-processing-n:127.0.0.1:40953_solr) 
[n:127.0.0.1:40953_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/halfcollection/state.json] for collection [halfcollection] 
has occurred - updating... (live nodes size: [4])
   [junit4]   2> 373289 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 373291 INFO  (qtp89624268-1579) [n:127.0.0.1:40953_solr 
c:halfcollection s:shard2 r:core_node1 x:halfcollection_shard2_replica1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=halfcollection_shard2_replica1&action=CREATE&numShards=2&collection=halfcollection&shard=shard2&wt=javabin&version=2}
 status=0 QTime=1444
   [junit4]   2> 373292 ERROR 
(OverseerThreadFactory-656-thread-2-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerCollectionMessageHandler Cleaning 
up collection [halfcollection].
   [junit4]   2> 373292 INFO  
(OverseerThreadFactory-656-thread-2-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing 
Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 373296 INFO  (qtp89624268-1558) [n:127.0.0.1:40953_solr    ] 
o.a.s.c.SolrCore [halfcollection_shard2_replica1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@5cacd77f
   [junit4]   2> 373310 INFO  (qtp89624268-1558) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:53630/data/halfcollection/core_node1/data/index
   [junit4]   2> 373313 INFO  (qtp89624268-1558) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:53630/data/halfcollection/core_node1/data/snapshot_metadata
   [junit4]   2> 373313 INFO  (qtp89624268-1558) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:53630/data/halfcollection/core_node1/data
   [junit4]   2> 373326 INFO  (IPC Server handler 7 on 53630) [    ] 
BlockStateChange BLOCK* addToInvalidates: blk_1073741827_1003 127.0.0.1:40732 
127.0.0.1:42118 
   [junit4]   2> 373345 INFO  (qtp89624268-1558) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=halfcollection_shard2_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=50
   [junit4]   2> 374049 INFO  
(OverseerThreadFactory-656-thread-2-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts 
for failed create collection for [halfcollection]
   [junit4]   2> 374053 INFO  (qtp516410542-1588) [n:127.0.0.1:60566_solr    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 374894 INFO  
(org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@5cc2157e)
 [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:40732 to delete 
[blk_1073741827_1003]
   [junit4]   2> 375619 INFO  
(OverseerCollectionConfigSetProcessor-96914350840610826-127.0.0.1:60566_solr-n_0000000000)
 [n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 377894 INFO  
(org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@5cc2157e)
 [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:42118 to delete 
[blk_1073741827_1003]
   [junit4]   2> 404057 ERROR (qtp516410542-1588) [n:127.0.0.1:60566_solr    ] 
o.a.s.h.a.CollectionsHandler Timed out waiting for new collection's replicas to 
become ACTIVE with timeout=30
   [junit4]   2> 404057 INFO  (qtp516410542-1588) [n:127.0.0.1:60566_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=1&collection.configName=conf&name=halfcollection&action=CREATE&numShards=2&createNodeSet=127.0.0.1:48954_solr,127.0.0.1:40953_solr&wt=javabin&version=2}
 status=0 QTime=32448
   [junit4]   2> 404058 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testCreateShouldFailOnExistingCore-seed#[F6C1D2403434BA00])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreateShouldFailOnExistingCore
   [junit4]   2> 404141 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[F6C1D2403434BA00])
 [    ] o.a.s.SolrTestCaseJ4 ###Starting testMissingRequiredParameters
   [junit4]   2> 404143 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[F6C1D2403434BA00])
 [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 404148 INFO  (qtp516410542-1582) [n:127.0.0.1:60566_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params 
name=halfcollectionblocker2&action=DELETE&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 404152 INFO  
(OverseerThreadFactory-656-thread-3-processing-n:127.0.0.1:60566_solr) 
[n:127.0.0.1:60566_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing 
Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 404164 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr    ] 
o.a.s.c.SolrCore [halfcollection_shard1_replica1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@7eed0fd6
   [junit4]   2> 404193 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:53630/data/halfcollectionblocker2/core_node1/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004/index
   [junit4]   2> 404194 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:53630/data/halfcollectionblocker2/core_node1/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004/snapshot_metadata
   [junit4]   2> 404194 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.h.HdfsDirectory Closing hdfs directory 
hdfs://localhost:53630/data/halfcollectionblocker2/core_node1/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-004
   [junit4]   2> 404207 INFO  (IPC Server handler 6 on 53630) [    ] 
BlockStateChange BLOCK* addToInvalidates: blk_1073741826_1002 127.0.0.1:40732 
127.0.0.1:42118 
   [junit4]   2> 404213 INFO  (qtp89624268-1557) [n:127.0.0.1:40953_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=halfcollection_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=49
   [junit4]   2> 404896 INFO 

[...truncated too long message...]

]-worker) [    ] o.m.log Stopped 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 555172 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-002/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-002/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:53630) [    ] o.a.h.h.s.d.DataNode 
BPOfferService for Block pool BP-575310218-10.41.0.5-1478795638828 (Datanode 
Uuid 6c5428c7-2126-40d8-82f5-5abb1bf54001) service to localhost/127.0.0.1:53630 
interrupted
   [junit4]   2> 555172 WARN  (DataNode: 
[[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-002/hdfsBaseDir/data/data1/,
 
[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001/tempDir-002/hdfsBaseDir/data/data2/]]
  heartbeating to localhost/127.0.0.1:53630) [    ] o.a.h.h.s.d.DataNode Ending 
block pool service for: Block pool BP-575310218-10.41.0.5-1478795638828 
(Datanode Uuid 6c5428c7-2126-40d8-82f5-5abb1bf54001) service to 
localhost/127.0.0.1:53630
   [junit4]   2> 555236 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 555240 ERROR 
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2>        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2>        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:52)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2>        ... 54 more
   [junit4]   2> 555254 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.c.ZkTestServer connecting to 127.0.0.1:47793 47793
   [junit4]   2> 555255 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[F6C1D2403434BA00]-worker) [    
] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_F6C1D2403434BA00-001
   [junit4]   2> Nov 10, 2016 4:37:26 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, 
docValues:{}, maxPointsInLeafNode=1097, maxMBSortInHeap=5.053770508665871, 
sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=en-CA, 
timezone=Brazil/East
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_102 (64-bit)/cpus=4,threads=5,free=388903744,total=493355008
   [junit4]   2> NOTE: All tests run in this JVM: [TestTestInjection, 
TestRandomFaceting, SolrRequestParserTest, CurrencyFieldOpenExchangeTest, 
TestShortCircuitedRequests, MultiThreadedOCPTest, SolrCLIZkUtilsTest, 
TestSerializedLuceneMatchVersion, TestCloudPivotFacet, TestSolrCloudSnapshots, 
SolrCoreTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [58/651 (1!)] on J0 in 217.46s, 20 tests, 1 failure, 2 
errors <<< FAILURES!

[...truncated 62569 lines...]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to