Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/291/

5 tests failed.
FAILED:  
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchAddsWithDelete

Error Message:
Timeout while trying to assert number of documents @ target_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ 
target_collection
        at 
__randomizedtesting.SeedInfo.seed([9D4FB6EDFC038CA9:EA55CF49541A8D85]:0)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:271)
        at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchAddsWithDelete(CdcrReplicationDistributedZkTest.java:532)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: expected:<59> but was:<0>
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.junit.Assert.assertEquals(Assert.java:456)
        at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:262)
        ... 42 more


FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:35230/l_/q/c8n_1x2_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Timeout 
occured while waiting response from server at: 
http://127.0.0.1:35230/l_/q/c8n_1x2_shard1_replica1
        at 
__randomizedtesting.SeedInfo.seed([9D4FB6EDFC038CA9:151B893752FFE151]:0)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:805)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1238)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:609)
        at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:595)
        at 
org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:357)
        at 
org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:125)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: 
http://127.0.0.1:35230/l_/q/c8n_1x2_shard1_replica1
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:621)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:780)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
        ... 10 more


FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv

Error Message:
java.lang.RuntimeException: Error from server at 
http://127.0.0.1:43728/solr/test_col: Async exception during distributed 
update: Error from server at 
http://127.0.0.1:60007/solr/test_col_shard1_replica2: Server Error    request: 
http://127.0.0.1:60007/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A43728%2Fsolr%2Ftest_col_shard4_replica1%2F&wt=javabin&version=2
 Remote error message: Failed synchronous update on shard StdNode: 
http://127.0.0.1:57618/solr/test_col_shard1_replica1/ update: 
org.apache.solr.client.solrj.request.UpdateRequest@2748274d

Stack Trace:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error from 
server at http://127.0.0.1:43728/solr/test_col: Async exception during 
distributed update: Error from server at 
http://127.0.0.1:60007/solr/test_col_shard1_replica2: Server Error



request: 
http://127.0.0.1:60007/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A43728%2Fsolr%2Ftest_col_shard4_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: 
http://127.0.0.1:57618/solr/test_col_shard1_replica1/ update: 
org.apache.solr.client.solrj.request.UpdateRequest@2748274d
        at 
__randomizedtesting.SeedInfo.seed([9D4FB6EDFC038CA9:AB5BD4AB765EB6B8]:0)
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:281)
        at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv(TestStressCloudBlindAtomicUpdates.java:193)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error from server at 
http://127.0.0.1:43728/solr/test_col: Async exception during distributed 
update: Error from server at 
http://127.0.0.1:60007/solr/test_col_shard1_replica2: Server Error



request: 
http://127.0.0.1:60007/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A43728%2Fsolr%2Ftest_col_shard4_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: 
http://127.0.0.1:57618/solr/test_col_shard1_replica1/ update: 
org.apache.solr.client.solrj.request.UpdateRequest@2748274d
        at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:409)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:43728/solr/test_col: Async exception during 
distributed update: Error from server at 
http://127.0.0.1:60007/solr/test_col_shard1_replica2: Server Error



request: 
http://127.0.0.1:60007/solr/test_col_shard1_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F127.0.0.1%3A43728%2Fsolr%2Ftest_col_shard4_replica1%2F&wt=javabin&version=2
Remote error message: Failed synchronous update on shard StdNode: 
http://127.0.0.1:57618/solr/test_col_shard1_replica1/ update: 
org.apache.solr.client.solrj.request.UpdateRequest@2748274d
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.doRandomAtomicUpdate(TestStressCloudBlindAtomicUpdates.java:368)
        at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates$Worker.run(TestStressCloudBlindAtomicUpdates.java:397)
        ... 6 more


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:56913/_ft

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:56913/_ft
        at 
__randomizedtesting.SeedInfo.seed([9D4FB6EDFC038CA9:151B893752FFE151]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:621)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1358)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1109)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1042)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:220)
        at 
org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
        at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
        ... 51 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
__randomizedtesting.SeedInfo.seed([9D4FB6EDFC038CA9:151B893752FFE151]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at 
org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest.test(HdfsWriteToMultipleCollectionsTest.java:137)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12216 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/init-core-data-001
   [junit4]   2> 1979448 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 1979449 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1979449 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: 
/mh_bui/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1979490 WARN  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1979500 WARN  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1979505 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 1979514 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_56605_hdfs____8v4mjo/webapp
   [junit4]   2> 1980018 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56605
   [junit4]   2> 1980110 WARN  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1980111 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 1980120 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_53575_datanode____.iyujvo/webapp
   [junit4]   2> 1980476 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53575
   [junit4]   2> 1980518 WARN  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1980519 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 1980527 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_40637_datanode____wnh5vp/webapp
   [junit4]   2> 1980615 INFO  (IPC Server handler 3 on 35403) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-85f10f43-79f5-4781-b143-b635440e0b7f node 
DatanodeRegistration(127.0.0.1:52701, 
datanodeUuid=678558c5-3356-4b49-9cda-cdfbc43c0805, infoPort=60407, 
infoSecurePort=0, ipcPort=35281, 
storageInfo=lv=-56;cid=testClusterID;nsid=534316469;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1980615 INFO  (IPC Server handler 3 on 35403) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-ea9b2689-58b8-4cb7-8934-9e91040ee592 node 
DatanodeRegistration(127.0.0.1:52701, 
datanodeUuid=678558c5-3356-4b49-9cda-cdfbc43c0805, infoPort=60407, 
infoSecurePort=0, ipcPort=35281, 
storageInfo=lv=-56;cid=testClusterID;nsid=534316469;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1980943 INFO  
(SUITE-HdfsWriteToMultipleCollectionsTest-seed#[9D4FB6EDFC038CA9]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40637
   [junit4]   2> 1981058 INFO  (IPC Server handler 8 on 35403) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-b748533c-c26f-4550-937e-3e3bd4535912 node 
DatanodeRegistration(127.0.0.1:40636, 
datanodeUuid=01a70f14-e1c4-46aa-81f7-f1e1bd87b18c, infoPort=54402, 
infoSecurePort=0, ipcPort=37807, 
storageInfo=lv=-56;cid=testClusterID;nsid=534316469;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 1981058 INFO  (IPC Server handler 8 on 35403) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-79100624-ba46-49d3-a373-0f51864c8ad5 node 
DatanodeRegistration(127.0.0.1:40636, 
datanodeUuid=01a70f14-e1c4-46aa-81f7-f1e1bd87b18c, infoPort=54402, 
infoSecurePort=0, ipcPort=37807, 
storageInfo=lv=-56;cid=testClusterID;nsid=534316469;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1981260 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1981261 INFO  (Thread-8004) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1981261 INFO  (Thread-8004) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1981361 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:54666
   [junit4]   2> 1981375 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1981376 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1981377 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1981377 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1981378 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1981378 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1981379 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1981380 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1981380 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1981381 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1981381 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1981452 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/control-001/cores/collection1
   [junit4]   2> 1981454 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1981454 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@252ba6ce{/mh_bui,null,AVAILABLE}
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@380e4c56{HTTP/1.1,[http/1.1]}{127.0.0.1:60431}
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.Server Started @1984505ms
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:35403/hdfs__localhost_35403__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001_tempDir-002_control_data,
 hostContext=/mh_bui, hostPort=60431, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/control-001/cores}
   [junit4]   2> 1981455 ERROR 
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.5.0
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1981455 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-02-22T14:40:38.364Z
   [junit4]   2> 1981457 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1981457 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/control-001/solr.xml
   [junit4]   2> 1981462 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1981463 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54666/solr
   [junit4]   2> 1981476 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1981476 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:60431_mh_bui
   [junit4]   2> 1981477 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.Overseer Overseer 
(id=97502785579974660-127.0.0.1:60431_mh_bui-n_0000000000) starting
   [junit4]   2> 1981485 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:60431_mh_bui
   [junit4]   2> 1981486 INFO  
(zkCallback-1700-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1981606 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/control-001/cores
   [junit4]   2> 1981606 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 1981614 INFO  
(OverseerStateUpdate-97502785579974660-127.0.0.1:60431_mh_bui-n_0000000000) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 1981652 WARN  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] o.a.s.c.Config 
Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> 
instead.
   [junit4]   2> 1981653 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1981667 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1981724 WARN  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] default search field in schema is text. 
WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1981725 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1981736 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 1981736 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:35403/solr_hdfs_home
   [junit4]   2> 1981736 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1981737 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/control-001/cores/collection1],
 
dataDir=[hdfs://localhost:35403/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1981737 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13ae72b3
   [junit4]   2> 1981737 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:35403/solr_hdfs_home/control_collection/core_node1/data/snapshot_metadata
   [junit4]   2> 1981743 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1981743 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1981743 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1981757 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1981760 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:35403/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1981776 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:35403/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1981781 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1981781 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1981781 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1981785 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1981785 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=29, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=28.2001953125, 
floorSegmentMB=1.2451171875, forceMergeDeletesPctAllowed=9.986173529396838, 
segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1981892 INFO  (IPC Server handler 3 on 35403) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-85f10f43-79f5-4781-b143-b635440e0b7f:NORMAL:127.0.0.1:52701|RBW],
 
ReplicaUC[[DISK]DS-b748533c-c26f-4550-937e-3e3bd4535912:NORMAL:127.0.0.1:40636|FINALIZED]]}
 size 0
   [junit4]   2> 1981893 INFO  (IPC Server handler 4 on 35403) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52701 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-85f10f43-79f5-4781-b143-b635440e0b7f:NORMAL:127.0.0.1:52701|RBW],
 
ReplicaUC[[DISK]DS-b748533c-c26f-4550-937e-3e3bd4535912:NORMAL:127.0.0.1:40636|FINALIZED]]}
 size 0
   [junit4]   2> 1981908 WARN  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1981961 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1981961 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1981961 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1981968 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1981969 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1981978 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=42, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=29.859375, 
floorSegmentMB=1.076171875, forceMergeDeletesPctAllowed=19.129815715157427, 
segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1981990 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@5208efc5[collection1] main]
   [junit4]   2> 1981991 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1981991 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1981991 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1982013 INFO  
(searcherExecutor-5473-thread-1-processing-n:127.0.0.1:60431_mh_bui 
x:collection1 c:control_collection) [n:127.0.0.1:60431_mh_bui 
c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered 
new searcher Searcher@5208efc5[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1982016 INFO  
(coreLoadExecutor-5472-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui c:control_collection   x:collection1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1560044570070220800
   [junit4]   2> 1982029 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1982030 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54666/solr ready
   [junit4]   2> 1982030 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 1982030 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1982031 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas 
found to continue.
   [junit4]   2> 1982031 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new 
leader - try and sync
   [junit4]   2> 1982031 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:60431/mh_bui/collection1/
   [junit4]   2> 1982031 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync 
replicas to me
   [junit4]   2> 1982031 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:60431/mh_bui/collection1/ has no replicas
   [junit4]   2> 1982033 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new 
leader: http://127.0.0.1:60431/mh_bui/collection1/ shard1
   [junit4]   2> 1982084 INFO  
(coreZkRegister-5465-thread-1-processing-n:127.0.0.1:60431_mh_bui x:collection1 
c:control_collection) [n:127.0.0.1:60431_mh_bui c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery 
necessary
   [junit4]   2> 1982117 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-1-001/cores/collection1
   [junit4]   2> 1982118 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-1-001
   [junit4]   2> 1982118 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3ebddd93{/mh_bui,null,AVAILABLE}
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@34dc1668{HTTP/1.1,[http/1.1]}{127.0.0.1:50107}
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.Server Started @1985171ms
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:35403/hdfs__localhost_35403__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001_tempDir-002_jetty1,
 solrconfig=solrconfig.xml, hostContext=/mh_bui, hostPort=50107, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-1-001/cores}
   [junit4]   2> 1982120 ERROR 
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.5.0
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1982120 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-02-22T14:40:39.029Z
   [junit4]   2> 1982122 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1982123 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-1-001/solr.xml
   [junit4]   2> 1982127 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1982128 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54666/solr
   [junit4]   2> 1982134 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1982135 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1982137 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:50107_mh_bui
   [junit4]   2> 1982138 INFO  (zkCallback-1704-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1982138 INFO  
(zkCallback-1700-thread-1-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1982153 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1982197 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-1-001/cores
   [junit4]   2> 1982197 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 1982202 INFO  
(OverseerStateUpdate-97502785579974660-127.0.0.1:60431_mh_bui-n_0000000000) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 1982303 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1983215 WARN  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.c.Config 
Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> 
instead.
   [junit4]   2> 1983216 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1983225 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 1983272 WARN  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] default search field in schema is text. WARNING: Deprecated, 
please use 'df' on request instead.
   [junit4]   2> 1983273 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1983296 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1983297 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:35403/solr_hdfs_home
   [junit4]   2> 1983297 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1983297 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-1-001/cores/collection1],
 dataDir=[hdfs://localhost:35403/solr_hdfs_home/collection1/core_node1/data/]
   [junit4]   2> 1983297 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13ae72b3
   [junit4]   2> 1983298 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:35403/solr_hdfs_home/collection1/core_node1/data/snapshot_metadata
   [junit4]   2> 1983305 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1983305 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1983305 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1983308 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1983309 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:35403/solr_hdfs_home/collection1/core_node1/data
   [junit4]   2> 1983323 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:35403/solr_hdfs_home/collection1/core_node1/data/index
   [junit4]   2> 1983328 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1983329 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1983329 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1983334 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1983335 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=29, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=28.2001953125, 
floorSegmentMB=1.2451171875, forceMergeDeletesPctAllowed=9.986173529396838, 
segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1983362 INFO  (IPC Server handler 1 on 35403) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52701 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-b748533c-c26f-4550-937e-3e3bd4535912:NORMAL:127.0.0.1:40636|RBW],
 
ReplicaUC[[DISK]DS-ea9b2689-58b8-4cb7-8934-9e91040ee592:NORMAL:127.0.0.1:52701|FINALIZED]]}
 size 0
   [junit4]   2> 1983362 INFO  (IPC Server handler 0 on 35403) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:40636 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-ea9b2689-58b8-4cb7-8934-9e91040ee592:NORMAL:127.0.0.1:52701|FINALIZED],
 
ReplicaUC[[DISK]DS-79100624-ba46-49d3-a373-0f51864c8ad5:NORMAL:127.0.0.1:40636|FINALIZED]]}
 size 0
   [junit4]   2> 1983375 WARN  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1983409 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1983409 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1983409 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.u.HdfsUpdateLog 
Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1983418 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1983418 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1983418 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=42, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=29.859375, 
floorSegmentMB=1.076171875, forceMergeDeletesPctAllowed=19.129815715157427, 
segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1983425 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@658a43b4[collection1] main]
   [junit4]   2> 1983431 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1983432 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1983432 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1983433 INFO  
(searcherExecutor-5484-thread-1-processing-n:127.0.0.1:50107_mh_bui 
x:collection1 c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1   
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@658a43b4[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1983435 INFO  
(coreLoadExecutor-5483-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui c:collection1   x:collection1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1560044571558150144
   [junit4]   2> 1983449 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to 
continue.
   [junit4]   2> 1983449 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try 
and sync
   [junit4]   2> 1983449 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:50107/mh_bui/collection1/
   [junit4]   2> 1983449 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1983449 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:50107/mh_bui/collection1/ 
has no replicas
   [junit4]   2> 1983452 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:50107/mh_bui/collection1/ shard1
   [junit4]   2> 1983553 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1983596 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-2-001/cores/collection1
   [junit4]   2> 1983596 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-2-001
   [junit4]   2> 1983597 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@22f57401{/mh_bui,null,AVAILABLE}
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@2414e96b{HTTP/1.1,[http/1.1]}{127.0.0.1:38618}
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.e.j.s.Server Started @1986652ms
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:35403/hdfs__localhost_35403__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_checkout_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001_tempDir-002_jetty2,
 solrconfig=solrconfig.xml, hostContext=/mh_bui, hostPort=38618, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-2-001/cores}
   [junit4]   2> 1983602 ERROR 
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.5.0
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1983602 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1983603 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-02-22T14:40:40.512Z
   [junit4]   2> 1983604 INFO  
(coreZkRegister-5478-thread-1-processing-n:127.0.0.1:50107_mh_bui x:collection1 
c:collection1) [n:127.0.0.1:50107_mh_bui c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1983606 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1983606 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-2-001/solr.xml
   [junit4]   2> 1983611 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1983612 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54666/solr
   [junit4]   2> 1983643 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 1983644 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1983645 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38618_mh_bui
   [junit4]   2> 1983646 INFO  (zkCallback-1704-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1983647 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1983647 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1983647 INFO  
(zkCallback-1700-thread-2-processing-n:127.0.0.1:60431_mh_bui) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1983654 INFO  
(zkCallback-1716-thread-1-processing-n:127.0.0.1:38618_mh_bui) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (3)
   [junit4]   2> 1983726 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsWriteToMultipleCollectionsTest_9D4FB6EDFC038CA9-001/shard-2-001/cores
   [junit4]   2> 1983726 INFO  
(TEST-HdfsWriteToMultipleCollectionsTest.test-seed#[9D4FB6EDFC038CA9]) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.CorePropertiesLocator Cores are: 
[collection1]
   [junit4]   2> 1983741 INFO  
(OverseerStateUpdate-97502785579974660-127.0.0.1:60431_mh_bui-n_0000000000) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 1983842 WARN  
(OverseerStateUpdate-97502785579974660-127.0.0.1:60431_mh_bui-n_0000000000) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.Overseer Bad version writing to ZK using 
compare-and-set, will force refresh cluster state: KeeperErrorCode = BadVersion 
for /collections/collection1/state.json
   [junit4]   2> 1983844 INFO  
(zkCallback-1716-thread-2-processing-n:127.0.0.1:38618_mh_bui) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1983845 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1983845 INFO  
(OverseerStateUpdate-97502785579974660-127.0.0.1:60431_mh_bui-n_0000000000) 
[n:127.0.0.1:60431_mh_bui    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 1983846 INFO  
(zkCallback-1716-thread-2-processing-n:127.0.0.1:38618_mh_bui) 
[n:127.0.0.1:38618_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1983846 INFO  
(zkCallback-1710-thread-1-processing-n:127.0.0.1:50107_mh_bui) 
[n:127.0.0.1:50107_mh_bui    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1984760 

[...truncated too long message...]

quired.java:53)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2>        ... 53 more
   [junit4]   2> 4697106 INFO  
(SUITE-StressHdfsTest-seed#[9D4FB6EDFC038CA9]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.StressHdfsTest_9D4FB6EDFC038CA9-001
   [junit4]   2> Feb 22, 2017 3:25:54 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{range_facet_l_dv=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 multiDefault=PostingsFormat(name=LuceneFixedGap), intDefault=FSTOrd50, 
id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 range_facet_i_dv=FSTOrd50, text=Lucene50(blocksize=128), 
range_facet_l=FSTOrd50, timestamp=FSTOrd50, txt_t=Lucene50(blocksize=128)}, 
docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), 
_version_=DocValuesFormat(name=Direct), 
range_facet_i_dv=DocValuesFormat(name=Direct), 
intDvoDefault=DocValuesFormat(name=Asserting), 
timestamp=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=481, 
maxMBSortInHeap=7.604129915618249, 
sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=es-DO, 
timezone=Etc/GMT+7
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 
1.8.0_121 (64-bit)/cpus=4,threads=1,free=159502544,total=535298048
   [junit4]   2> NOTE: All tests run in this JVM: [SuggesterWFSTTest, 
TestIndexingPerformance, QueryParsingTest, DateRangeFieldTest, 
HighlighterMaxOffsetTest, TestWriterPerf, TestExtendedDismaxParser, 
BlobRepositoryCloudTest, SolrCoreCheckLockOnStartupTest, 
TestSortByMinMaxFunction, DistributedQueryComponentCustomSortTest, 
DocValuesNotIndexedTest, TestAnalyzedSuggestions, BasicDistributedZk2Test, 
TestCoreAdminApis, CircularListTest, FacetPivotSmallTest, 
TestCustomDocTransformer, TestCopyFieldCollectionResource, 
HdfsDirectoryFactoryTest, DirectUpdateHandlerTest, TestStressReorder, 
BlockJoinFacetRandomTest, TimeZoneUtilsTest, TestConfigSetsAPI, 
TestRandomFlRTGCloud, FieldAnalysisRequestHandlerTest, TestManagedSchemaAPI, 
TestPerFieldSimilarityWithDefaultOverride, TestInPlaceUpdatesStandalone, 
DeleteNodeTest, OverseerRolesTest, TestPostingsSolrHighlighter, 
DirectUpdateHandlerOptimizeTest, TestHashQParserPlugin, 
TestSubQueryTransformerDistrib, ReplaceNodeTest, TestRestManager, 
TestDFRSimilarityFactory, FieldMutatingUpdateProcessorTest, 
HdfsRecoverLeaseTest, TestSort, RequestLoggingTest, TestRawResponseWriter, 
CloudMLTQParserTest, SortSpecParsingTest, CoreMergeIndexesAdminHandlerTest, 
TestFileDictionaryLookup, TermVectorComponentDistributedTest, SmileWriterTest, 
MissingSegmentRecoveryTest, BlockCacheTest, TestXmlQParser, 
TestDFISimilarityFactory, TestSolrDeletionPolicy1, TestRecovery, 
LukeRequestHandlerTest, LeaderElectionTest, TestSolrConfigHandlerCloud, 
ShowFileRequestHandlerTest, TestFieldSortValues, HdfsSyncSliceTest, 
TestSolrCloudWithHadoopAuthPlugin, HardAutoCommitTest, 
DocExpirationUpdateProcessorFactoryTest, ConjunctionSolrSpellCheckerTest, 
StatelessScriptUpdateProcessorFactoryTest, SimpleFacetsTest, 
TestNamedUpdateProcessors, TestGroupingSearch, TestSolr4Spatial2, 
SolrGraphiteReporterTest, TestApiFramework, OverseerTest, 
DistribJoinFromCollectionTest, TestDistribIDF, SOLR749Test, 
OverriddenZkACLAndCredentialsProvidersTest, NoCacheHeaderTest, 
AliasIntegrationTest, PreAnalyzedFieldTest, SuggesterTest, UtilsToolTest, 
BitVectorTest, TestConfig, ShufflingReplicaListTransformerTest, 
TestShardHandlerFactory, TestHdfsBackupRestoreCore, TestReRankQParserPlugin, 
ManagedSchemaRoundRobinCloudTest, TestStandardQParsers, 
BinaryUpdateRequestHandlerTest, TestInitParams, TestPivotHelperCode, 
TestDistributedMissingSort, TestSearcherReuse, TestQueryWrapperFilter, 
DOMUtilTest, TestMergePolicyConfig, DocValuesMultiTest, 
LeaderInitiatedRecoveryOnCommitTest, PreAnalyzedUpdateProcessorTest, 
TestCryptoKeys, BadIndexSchemaTest, TestSolrXml, TestPhraseSuggestions, 
CoreAdminHandlerTest, TestNumericTerms64, SuggesterTSTTest, TestDynamicLoading, 
TestJmxMonitoredMap, TestCollationField, UnloadDistributedZkTest, 
CollectionsAPIDistributedZkTest, SyncSliceTest, LeaderElectionIntegrationTest, 
ShardRoutingTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, 
TestRandomFaceting, TestZkChroot, TestDistributedSearch, AutoCommitTest, 
TestDistributedGrouping, TestRealTimeGet, TestJoin, TestReload, 
TestSolr4Spatial, StatsComponentTest, TestFunctionQuery, 
BasicFunctionalityTest, TestIndexSearcher, CurrencyFieldOpenExchangeTest, 
DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, 
SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, 
SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, 
SolrRequestParserTest, TestFoldingMultitermQuery, TestTrie, 
SchemaVersionSpecificBehaviorTest, TestAtomicUpdateErrorCases, 
TestWordDelimiterFilterFactory, QueryEqualityTest, TestRemoteStreaming, 
DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, 
TestSurroundQueryParser, StandardRequestHandlerTest, 
DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, 
DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, TestOmitPositions, 
FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, 
PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, RequiredFieldsTest, 
TestSolrQueryParser, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, 
LoggingHandlerTest, SolrPluginUtilsTest, ReturnFieldsTest, 
UpdateRequestProcessorFactoryTest, JSONWriterTest, TestCSVResponseWriter, 
MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, 
TestPartialUpdateDeduplication, PingRequestHandlerTest, CSVRequestHandlerTest, 
TestBinaryResponseWriter, TestLFUCache, TestQuerySenderListener, 
UpdateParamsTest, AlternateDirectoryTest, ResponseLogComponentTest, 
TestSolrDeletionPolicy2, TestDocSet, TestSearchPerf, MinimalSchemaTest, 
DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SynonymTokenizerTest, 
EchoParamsTest, TestLMJelinekMercerSimilarityFactory, 
OpenExchangeRatesOrgProviderTest, PluginInfoTest, TestSystemIdResolver, 
PrimUtilsTest, SpellingQueryConverterTest, TestSolrJ, TestLRUCache, TestUtils, 
TestDocumentBuilder, FileUtilsTest, TestRTGBase, CursorPagingTest, 
SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, 
TestTolerantSearch, TestPathTrie, TestEmbeddedSolrServerConstructors, 
TestJettySolrRunner, ConnectionReuseTest, AssignTest, 
AsyncCallRequestStatusResponseTest, BaseCdcrDistributedZkTest, 
CdcrReplicationHandlerTest, ConfigSetsAPITest, CustomCollectionTest, 
DeleteLastCustomShardedReplicaTest, DeleteStatusTest, DistribCursorPagingTest, 
DistributedQueueTest, DistributedVersionInfoTest, LeaderElectionContextKeyTest, 
LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, 
LeaderInitiatedRecoveryOnShardRestartTest, MigrateRouteKeyTest, 
MultiThreadedOCPTest, OutOfBoxZkACLAndCredentialsProvidersTest, 
OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, 
OverseerTaskQueueTest, PeerSyncReplicationTest, RemoteQueryErrorTest, 
RollingRestartTest, TestLeaderElectionZkExpiry, 
TestLeaderInitiatedRecoveryThread, TestMiniSolrCloudClusterSSL, 
TestRebalanceLeaders, TestStressLiveNodes, 
VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, 
HdfsBasicDistributedZkTest, HdfsRecoveryZkTest, HdfsThreadLeakTest, 
StressHdfsTest]
   [junit4] Completed [620/695 (5!)] on J2 in 185.80s, 1 test, 1 error <<< 
FAILURES!

[...truncated 62555 lines...]
changes-to-html:
    [mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/docs/changes
     [exec] Section 'Optimizations' appears more than once under release 
'6.5.0' at 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/site/changes/changes2html.pl
 line 136.

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/build.xml:783:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/build.xml:101:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build.xml:654:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build.xml:629:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/common-build.xml:2518:
 exec returned: 25

Total time: 267 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to