[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1349 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/1349/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=15640, 
name=updateExecutor-2641-thread-4, state=RUNNABLE, 
group=TGRP-ChaosMonkeySafeLeaderTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=15640, name=updateExecutor-2641-thread-4, 
state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at 
__randomizedtesting.SeedInfo.seed([79F11EE92CC8BDFD:F1A521338234D005]:0)
Caused by: org.apache.solr.common.SolrException: Replica: 
http://127.0.0.1:57707/collection1/ should have been marked under leader 
initiated recovery in ZkController but wasn't.
        at __randomizedtesting.SeedInfo.seed([79F11EE92CC8BDFD]:0)
        at 
org.apache.solr.cloud.LeaderInitiatedRecoveryThread.run(LeaderInitiatedRecoveryThread.java:88)
        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddReplica

Error Message:
Error from server at http://127.0.0.1:39567/solr: delete the collection time 
out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:39567/solr: delete the collection time out:180s
        at 
__randomizedtesting.SeedInfo.seed([79F11EE92CC8BDFD:F9D17BC73D8B555B]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:237)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1130)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at 
org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:442)
        at 
org.apache.solr.cloud.autoscaling.TestPolicyCloud.after(TestPolicyCloud.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:965)
        at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.TestPolicyCloud.testCreateCollectionAddShardUsingPolicy

Error Message:
Error from server at http://127.0.0.1:39567/solr: Could not find collection : 
testCreateCollectionAddReplica

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:39567/solr: Could not find collection : 
testCreateCollectionAddReplica
        at 
__randomizedtesting.SeedInfo.seed([79F11EE92CC8BDFD:B86D2C40EDF0403E]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:610)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:237)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:459)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:389)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1130)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:177)
        at 
org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:442)
        at 
org.apache.solr.cloud.autoscaling.TestPolicyCloud.after(TestPolicyCloud.java:63)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:965)
        at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12030 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/init-core-data-001
   [junit4]   2> 1411180 WARN  
(SUITE-ChaosMonkeySafeLeaderTest-seed#[79F11EE92CC8BDFD]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 1411180 INFO  
(SUITE-ChaosMonkeySafeLeaderTest-seed#[79F11EE92CC8BDFD]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using TrieFields
   [junit4]   2> 1411181 INFO  
(SUITE-ChaosMonkeySafeLeaderTest-seed#[79F11EE92CC8BDFD]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1411181 INFO  
(SUITE-ChaosMonkeySafeLeaderTest-seed#[79F11EE92CC8BDFD]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1411182 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1411182 INFO  (Thread-3028) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1411182 INFO  (Thread-3028) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1411184 ERROR (Thread-3028) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1411282 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:42291
   [junit4]   2> 1411310 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1411316 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1411321 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1411325 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1411328 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1411332 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1411336 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1411339 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1411342 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1411346 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1411349 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1411354 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly 
asked otherwise
   [junit4]   2> 1411467 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/control-001/cores/collection1
   [junit4]   2> 1411468 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1411468 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1ecd353f{/,null,AVAILABLE}
   [junit4]   2> 1411469 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@459a564f{HTTP/1.1,[http/1.1]}{127.0.0.1:43706}
   [junit4]   2> 1411469 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.Server Started @1415653ms
   [junit4]   2> 1411469 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/tempDir-001/control/data,
 hostContext=/, hostPort=43706, 
coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/control-001/cores}
   [junit4]   2> 1411469 ERROR 
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1411470 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.0.0
   [junit4]   2> 1411470 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1411470 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1411470 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-08T08:06:21.330Z
   [junit4]   2> 1411473 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1411473 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/control-001/solr.xml
   [junit4]   2> 1411476 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1411478 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42291/solr
   [junit4]   2> 1411503 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1411504 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:43706_
   [junit4]   2> 1411505 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.c.Overseer Overseer 
(id=98101440088834052-127.0.0.1:43706_-n_0000000000) starting
   [junit4]   2> 1411513 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:43706_
   [junit4]   2> 1411516 INFO  
(OverseerStateUpdate-98101440088834052-127.0.0.1:43706_-n_0000000000) 
[n:127.0.0.1:43706_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1411630 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1411632 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1411632 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1411634 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions 
underneath 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/control-001/cores
   [junit4]   2> 1411634 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:43706_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1411634 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 1411637 INFO  
(OverseerStateUpdate-98101440088834052-127.0.0.1:43706_-n_0000000000) 
[n:127.0.0.1:43706_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 1412662 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1412697 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 1412761 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1412779 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
control_collection, trusted=true
   [junit4]   2> 1412779 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.control_collection.shard1.core_node1' (registry 
'solr.core.control_collection.shard1.core_node1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1412779 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1412779 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/control-001/cores/collection1],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/control-001/cores/collection1/data/]
   [junit4]   2> 1412781 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=20, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=41.513671875, 
floorSegmentMB=1.58984375, forceMergeDeletesPctAllowed=20.14306085486076, 
segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1412785 WARN  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1412817 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1412817 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1412818 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.u.CommitTracker 
Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 1412818 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1412819 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=32, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=92.6796875, 
floorSegmentMB=1.8974609375, forceMergeDeletesPctAllowed=12.841826748069575, 
segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.21196876632300787
   [junit4]   2> 1412819 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.s.SolrIndexSearcher Opening [Searcher@6dc15ada[collection1] main]
   [junit4]   2> 1412821 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1412821 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1412821 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1412822 INFO  
(coreLoadExecutor-7464-thread-1-processing-n:127.0.0.1:43706_) 
[n:127.0.0.1:43706_ c:control_collection   x:collection1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1569623043139960832
   [junit4]   2> 1412822 INFO  
(searcherExecutor-7465-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection   x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@6dc15ada[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1412828 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas 
found to continue.
   [junit4]   2> 1412829 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new 
leader - try and sync
   [junit4]   2> 1412829 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:43706/collection1/
   [junit4]   2> 1412829 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync 
replicas to me
   [junit4]   2> 1412829 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:43706/collection1/ has no replicas
   [junit4]   2> 1412829 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Found all 
replicas participating in election, clear LIR
   [junit4]   2> 1412834 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new 
leader: http://127.0.0.1:43706/collection1/ shard1
   [junit4]   2> 1412989 INFO  
(coreZkRegister-7457-thread-1-processing-n:127.0.0.1:43706_ x:collection1 
c:control_collection) [n:127.0.0.1:43706_ c:control_collection s:shard1 
r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery 
necessary
   [junit4]   2> 1413158 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1413161 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42291/solr ready
   [junit4]   2> 1413161 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 1413161 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1413286 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001/cores/collection1
   [junit4]   2> 1413286 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001
 of type NRT
   [junit4]   2> 1413287 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1413288 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@3289860e{/,null,AVAILABLE}
   [junit4]   2> 1413288 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@2c2be622{HTTP/1.1,[http/1.1]}{127.0.0.1:38599}
   [junit4]   2> 1413288 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.Server Started @1417472ms
   [junit4]   2> 1413289 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/tempDir-001/jetty1,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=38599, 
coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001/cores}
   [junit4]   2> 1413289 ERROR 
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1413289 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.0.0
   [junit4]   2> 1413289 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1413289 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1413289 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-08T08:06:23.149Z
   [junit4]   2> 1413292 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1413292 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001/solr.xml
   [junit4]   2> 1413295 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1413297 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42291/solr
   [junit4]   2> 1413308 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1413311 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1413314 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:38599_
   [junit4]   2> 1413316 INFO  
(zkCallback-2634-thread-2-processing-n:127.0.0.1:43706_) [n:127.0.0.1:43706_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1413317 INFO  (zkCallback-2638-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1413317 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1413390 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1413393 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1413393 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1413395 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions 
underneath 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001/cores
   [junit4]   2> 1413395 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:38599_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1413396 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 1413398 INFO  
(OverseerStateUpdate-98101440088834052-127.0.0.1:43706_-n_0000000000) 
[n:127.0.0.1:43706_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 1413502 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1414425 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 7.0.0
   [junit4]   2> 1414442 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 1414530 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded 
schema test/1.6 with uniqueid field id
   [junit4]   2> 1414548 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
collection1, trusted=true
   [junit4]   2> 1414548 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.m.r.SolrJmxReporter 
JMX monitoring for 'solr.core.collection1.shard1.core_node1' (registry 
'solr.core.collection1.shard1.core_node1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1414548 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1414548 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001/cores/collection1],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1414550 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy 
RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: 
[TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=35, 
maxMergedSegmentMB=41.513671875, floorSegmentMB=1.58984375, 
forceMergeDeletesPctAllowed=20.14306085486076, segmentsPerTier=41.0, 
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1414554 WARN  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.c.RequestHandlers 
INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class 
= DumpRequestHandler,attributes = {initParams=a, name=/dump, 
class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1414586 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using 
UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1414586 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1414586 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard 
AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 1414586 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft 
AutoCommit: disabled
   [junit4]   2> 1414587 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy 
RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: 
[TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=44, 
maxMergedSegmentMB=92.6796875, floorSegmentMB=1.8974609375, 
forceMergeDeletesPctAllowed=12.841826748069575, segmentsPerTier=40.0, 
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21196876632300787
   [junit4]   2> 1414587 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher 
Opening [Searcher@478fc8a3[collection1] main]
   [junit4]   2> 1414589 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1414589 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1414590 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler 
Commits will be reserved for  10000
   [junit4]   2> 1414590 INFO  
(coreLoadExecutor-7475-thread-1-processing-n:127.0.0.1:38599_) 
[n:127.0.0.1:38599_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not 
find max version in index or recent updates, using new clock 1569623044993843200
   [junit4]   2> 1414590 INFO  
(searcherExecutor-7476-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1   x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@478fc8a3[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1414610 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to 
continue.
   [junit4]   2> 1414610 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try 
and sync
   [junit4]   2> 1414610 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:38599/collection1/
   [junit4]   2> 1414610 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1414610 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:38599/collection1/ has no 
replicas
   [junit4]   2> 1414610 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext Found all replicas 
participating in election, clear LIR
   [junit4]   2> 1414615 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:38599/collection1/ shard1
   [junit4]   2> 1414721 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1414767 INFO  
(coreZkRegister-7470-thread-1-processing-n:127.0.0.1:38599_ x:collection1 
c:collection1) [n:127.0.0.1:38599_ c:collection1 s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1414876 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1415040 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001/cores/collection1
   [junit4]   2> 1415040 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001
 of type NRT
   [junit4]   2> 1415041 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1415043 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1eeb4012{/,null,AVAILABLE}
   [junit4]   2> 1415043 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5d32260c{HTTP/1.1,[http/1.1]}{127.0.0.1:57707}
   [junit4]   2> 1415043 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.e.j.s.Server Started @1419228ms
   [junit4]   2> 1415043 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/tempDir-001/jetty2,
 solrconfig=solrconfig.xml, hostContext=/, hostPort=57707, 
coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001/cores}
   [junit4]   2> 1415044 ERROR 
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1415044 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
7.0.0
   [junit4]   2> 1415044 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1415044 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1415045 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-08T08:06:24.905Z
   [junit4]   2> 1415048 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1415048 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001/solr.xml
   [junit4]   2> 1415050 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1415052 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42291/solr
   [junit4]   2> 1415063 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 1415066 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1415068 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:57707_
   [junit4]   2> 1415070 INFO  (zkCallback-2638-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1415070 INFO  
(zkCallback-2634-thread-2-processing-n:127.0.0.1:43706_) [n:127.0.0.1:43706_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1415070 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1415071 INFO  
(zkCallback-2650-thread-1-processing-n:127.0.0.1:57707_) [n:127.0.0.1:57707_    
] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1415179 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1415181 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1415182 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1415183 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions 
underneath 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001/cores
   [junit4]   2> 1415183 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) 
[n:127.0.0.1:57707_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1415184 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 1415185 INFO  
(OverseerStateUpdate-98101440088834052-127.0.0.1:43706_-n_0000000000) 
[n:127.0.0.1:43706_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 1415289 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1415290 INFO  
(zkCallback-2650-thread-1-processing-n:127.0.0.1:57707_) [n:127.0.0.1:57707_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1416215 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 7.0.0
   [junit4]   2> 1416232 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 1416310 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded 
schema test/1.6 with uniqueid field id
   [junit4]   2> 1416330 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
collection1, trusted=true
   [junit4]   2> 1416330 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.m.r.SolrJmxReporter 
JMX monitoring for 'solr.core.collection1.shard1.core_node2' (registry 
'solr.core.collection1.shard1.core_node2') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@4b221af8
   [junit4]   2> 1416330 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1416330 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.c.SolrCore 
[[collection1] ] Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001/cores/collection1],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_79F11EE92CC8BDFD-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 1416332 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy 
RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: 
[TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=35, 
maxMergedSegmentMB=41.513671875, floorSegmentMB=1.58984375, 
forceMergeDeletesPctAllowed=20.14306085486076, segmentsPerTier=41.0, 
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1416335 WARN  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.c.RequestHandlers 
INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class 
= DumpRequestHandler,attributes = {initParams=a, name=/dump, 
class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1416368 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using 
UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1416368 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1416369 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard 
AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 1416369 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft 
AutoCommit: disabled
   [junit4]   2> 1416370 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy 
RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: 
[TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=44, 
maxMergedSegmentMB=92.6796875, floorSegmentMB=1.8974609375, 
forceMergeDeletesPctAllowed=12.841826748069575, segmentsPerTier=40.0, 
maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21196876632300787
   [junit4]   2> 1416370 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher 
Opening [Searcher@7a2df094[collection1] main]
   [junit4]   2> 1416371 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1416372 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1416372 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler 
Commits will be reserved for  10000
   [junit4]   2> 1416373 INFO  
(searcherExecutor-7487-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
c:collection1) [n:127.0.0.1:57707_ c:collection1   x:collection1] 
o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@7a2df094[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1416373 INFO  
(coreLoadExecutor-7486-thread-1-processing-n:127.0.0.1:57707_) 
[n:127.0.0.1:57707_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not 
find max version in index or recent updates, using new clock 1569623046863454208
   [junit4]   2> 1416377 INFO  
(coreZkRegister-7481-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
c:collection1) [n:127.0.0.1:57707_ c:collection1 s:shard1 r:core_node2 
x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 1416378 INFO  
(updateExecutor-2647-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 1416378 INFO  
(recoveryExecutor-2648-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. 
recoveringAfterStartup=true
   [junit4]   2> 1416378 INFO  
(recoveryExecutor-2648-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 1416378 INFO  
(recoveryExecutor-2648-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. 
core=[collection1]
   [junit4]   2> 1416378 INFO  
(recoveryExecutor-2648-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. 
FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 1416378 INFO  
(recoveryExecutor-2648-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core 
[collection1] as recovering, leader is [http://127.0.0.1:38599/collection1/] 
and I am [http://127.0.0.1:57707/collection1/]
   [junit4]   2> 1416382 INFO  
(recoveryExecutor-2648-thread-1-processing-n:127.0.0.1:57707_ x:collection1 
s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:57707_ c:collection1 s:shard1 
r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery 
command to [http://127.0.0.1:38599]; [WaitForState: 
action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:57707_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 1416384 INFO  (qtp2062388605-15428) [n:127.0.0.1:38599_    ] 
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: 
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, 
maxTime: 183 s
   [junit4]   2> 1416385 INFO  (qtp2062388605-15428) [n:127.0.0.1:38599_    ] 
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, 
shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? 
true, live=true, checkLive=true, currentState=down, localState=active, 
nodeName=127.0.0.1:57707_, coreNodeName=core_node2, 
onlyIfActiveCheckResult=false, nodeProps: 
core_node2:{"core":"collection1","base_url":"http://127.0.0.1:57707","node_name":"127.0.0.1:57707_","state":"down","type":"NRT"}
   [junit4]   2> 1416488 INFO  
(zkCallback-2644-thread-1-processing-n:127.0.0.1:38599_) [n:127.0.0.1:38599_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1416488 INFO  
(zkCallback-2650-thread-1-processing-n:127.0.0.1:57707_) [n:127.0.0.1:57707_    
] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/collection1/state.json] for collection [collection1] has 
occurred - updating... (live nodes size: [3])
   [junit4]   2> 1416711 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1416711 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Turning on auto soft commit: 1000
   [junit4]   2> 1416719 INFO  (qtp563693276-15390) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{deleteByQuery=*:* (-1569623047223115776)} 0 3
   [junit4]   2> 1416734 INFO  (qtp956463895-15456) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&_version_=-1569623047230455808&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{deleteByQuery=*:*
 (-1569623047230455808)} 0 1
   [junit4]   2> 1416734 INFO  (qtp2062388605-15430) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{deleteByQuery=*:* (-1569623047230455808)} 0 12
   [junit4]   2> 1416736 INFO  
(TEST-ChaosMonkeySafeLeaderTest.test-seed#[79F11EE92CC8BDFD]) [    ] 
o.a.s.c.ChaosMonkey monkey: starting
   [junit4]   2> 1416741 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-0 (1569623047248281600)]} 0 2
   [junit4]   2> 1416742 INFO  (qtp563693276-15391) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-0 (1569623047249330176)]} 0 1
   [junit4]   2> 1416750 INFO  (qtp956463895-15457) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-0
 (1569623047255621632)]} 0 0
   [junit4]   2> 1416751 INFO  (qtp2062388605-15431) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-0 (1569623047255621632)]} 0 4
   [junit4]   2> 1416754 INFO  (qtp956463895-15458) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-0
 (1569623047257718784)]} 0 0
   [junit4]   2> 1416754 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-1 (1569623047262961664)]} 0 0
   [junit4]   2> 1416754 INFO  (qtp2062388605-15432) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-0 (1569623047257718784)]} 0 5
   [junit4]   2> 1416755 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-1 (1569623047264010240)]} 0 0
   [junit4]   2> 1416757 INFO  (qtp956463895-15459) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-1
 (1569623047264010240)]} 0 0
   [junit4]   2> 1416757 INFO  (qtp2062388605-15433) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-1 (1569623047264010240)]} 0 2
   [junit4]   2> 1416762 INFO  (qtp956463895-15460) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-1
 (1569623047266107392)]} 0 0
   [junit4]   2> 1416763 INFO  (qtp2062388605-15426) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-1 (1569623047266107392)]} 0 6
   [junit4]   2> 1416763 INFO  (qtp563693276-15394) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-2 (1569623047272398848)]} 0 0
   [junit4]   2> 1416764 INFO  (qtp563693276-15387) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-2 (1569623047273447424)]} 0 0
   [junit4]   2> 1416766 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-2
 (1569623047273447424)]} 0 0
   [junit4]   2> 1416766 INFO  (qtp2062388605-15429) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-2 (1569623047273447424)]} 0 1
   [junit4]   2> 1416766 INFO  (qtp956463895-15454) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-2
 (1569623047274496000)]} 0 0
   [junit4]   2> 1416766 INFO  (qtp2062388605-15430) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-2 (1569623047274496000)]} 0 1
   [junit4]   2> 1416767 INFO  (qtp563693276-15390) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-0 (-1569623047276593152)]} 
0 0
   [junit4]   2> 1416767 INFO  (qtp563693276-15392) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[0-0 (-1569623047276593153)]} 
0 0
   [junit4]   2> 1416768 INFO  (qtp956463895-15456) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[1-0
 (-1569623047276593152)]} 0 0
   [junit4]   2> 1416768 INFO  (qtp2062388605-15431) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[1-0 (-1569623047276593152)]} 0 0
   [junit4]   2> 1416769 INFO  (qtp956463895-15457) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[0-0
 (-1569623047277641728)]} 0 0
   [junit4]   2> 1416769 INFO  (qtp563693276-15391) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-3 (1569623047278690304)]} 0 0
   [junit4]   2> 1416769 INFO  (qtp2062388605-15432) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[0-0 (-1569623047277641728)]} 0 0
   [junit4]   2> 1416770 INFO  (qtp956463895-15458) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-3
 (1569623047278690304)]} 0 0
   [junit4]   2> 1416770 INFO  (qtp563693276-15393) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-3 (1569623047279738880)]} 0 0
   [junit4]   2> 1416770 INFO  (qtp2062388605-15433) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-3 (1569623047278690304)]} 0 0
   [junit4]   2> 1416771 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-4 (1569623047280787456)]} 0 0
   [junit4]   2> 1416771 INFO  (qtp956463895-15459) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-3
 (1569623047280787456)]} 0 0
   [junit4]   2> 1416771 INFO  (qtp2062388605-15426) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-3 (1569623047280787456)]} 0 0
   [junit4]   2> 1416772 INFO  (qtp563693276-15394) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-4 (1569623047281836032)]} 0 0
   [junit4]   2> 1416772 INFO  (qtp956463895-15460) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-4
 (1569623047280787457)]} 0 0
   [junit4]   2> 1416772 INFO  (qtp2062388605-15429) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-4 (1569623047280787457)]} 0 0
   [junit4]   2> 1416773 INFO  (qtp563693276-15387) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-2 (-1569623047282884608)]} 
0 0
   [junit4]   2> 1416773 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-4
 (1569623047281836032)]} 0 0
   [junit4]   2> 1416773 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[1-2
 (-1569623047282884608)]} 0 0
   [junit4]   2> 1416773 INFO  (qtp2062388605-15430) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-4 (1569623047281836032)]} 0 0
   [junit4]   2> 1416774 INFO  (qtp2062388605-15431) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[1-2 (-1569623047282884608)]} 0 0
   [junit4]   2> 1416774 INFO  (qtp563693276-15387) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[0-2 (-1569623047283933184)]} 
0 0
   [junit4]   2> 1416774 INFO  (qtp563693276-15392) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-5 (1569623047283933185)]} 0 0
   [junit4]   2> 1416775 INFO  (qtp956463895-15454) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[0-2
 (-1569623047283933184)]} 0 0
   [junit4]   2> 1416775 INFO  (qtp2062388605-15432) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[0-2 (-1569623047283933184)]} 0 0
   [junit4]   2> 1416775 INFO  (qtp956463895-15456) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-5
 (1569623047284981760)]} 0 0
   [junit4]   2> 1416775 INFO  (qtp563693276-15391) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-5 (1569623047284981760)]} 0 0
   [junit4]   2> 1416776 INFO  (qtp2062388605-15433) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-5 (1569623047284981760)]} 0 0
   [junit4]   2> 1416776 INFO  (qtp563693276-15393) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-6 (1569623047286030336)]} 0 0
   [junit4]   2> 1416777 INFO  (qtp956463895-15457) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-5
 (1569623047286030336)]} 0 0
   [junit4]   2> 1416777 INFO  (qtp956463895-15458) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-6
 (1569623047287078912)]} 0 0
   [junit4]   2> 1416777 INFO  (qtp2062388605-15426) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-5 (1569623047286030336)]} 0 1
   [junit4]   2> 1416777 INFO  (qtp2062388605-15429) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-6 (1569623047287078912)]} 0 0
   [junit4]   2> 1416778 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-6 (1569623047288127488)]} 0 0
   [junit4]   2> 1416778 INFO  (qtp563693276-15394) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-6 (-1569623047288127489)]} 
0 0
   [junit4]   2> 1416778 INFO  (qtp956463895-15459) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-6
 (1569623047288127489)]} 0 0
   [junit4]   2> 1416779 INFO  (qtp2062388605-15431) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-6 (1569623047288127489)]} 0 0
   [junit4]   2> 1416779 INFO  (qtp956463895-15459) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[1-6
 (-1569623047288127488)]} 0 0
   [junit4]   2> 1416779 INFO  (qtp2062388605-15430) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[1-6 (-1569623047288127488)]} 0 1
   [junit4]   2> 1416780 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[0-6 (-1569623047290224640)]} 
0 0
   [junit4]   2> 1416780 INFO  (qtp956463895-15460) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[0-6
 (-1569623047290224640)]} 0 0
   [junit4]   2> 1416781 INFO  (qtp2062388605-15432) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[0-6 (-1569623047290224640)]} 0 0
   [junit4]   2> 1416781 INFO  (qtp563693276-15389) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-7 (1569623047291273216)]} 0 0
   [junit4]   2> 1416782 INFO  (qtp563693276-15392) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-7 (1569623047291273217)]} 0 0
   [junit4]   2> 1416782 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[0-7
 (1569623047292321792)]} 0 0
   [junit4]   2> 1416783 INFO  (qtp2062388605-15433) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[0-7 (1569623047292321792)]} 0 0
   [junit4]   2> 1416783 INFO  (qtp563693276-15392) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[0-7 (-1569623047293370368)]} 
0 0
   [junit4]   2> 1416784 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[0-7
 (-1569623047293370369)]} 0 0
   [junit4]   2> 1416784 INFO  (qtp956463895-15454) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-7
 (1569623047293370368)]} 0 0
   [junit4]   2> 1416784 INFO  (qtp2062388605-15429) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[0-7 (-1569623047293370369)]} 0 0
   [junit4]   2> 1416784 INFO  (qtp2062388605-15426) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-7 (1569623047293370368)]} 0 0
   [junit4]   2> 1416785 INFO  (qtp563693276-15392) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[0-8 (1569623047294418944)]} 0 0
   [junit4]   2> 1416785 INFO  (qtp563693276-15390) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-7 (-1569623047295467520)]} 
0 0
   [junit4]   2> 1416786 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{delete=[1-7
 (-1569623047296516096)]} 0 0
   [junit4]   2> 1416786 INFO  (qtp2062388605-15431) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{delete=[1-7 (-1569623047296516096)]} 0 0
   [junit4]   2> 1416787 INFO  (qtp563693276-15394) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-8 (1569623047297564672)]} 0 0
   [junit4]   2> 1416788 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-8
 (1569623047297564672)]} 0 0
   [junit4]   2> 1416788 INFO  (qtp2062388605-15430) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2}{add=[1-8 (1569623047297564672)]} 0 1
   [junit4]   2> 1416789 INFO  (qtp563693276-15394) [n:127.0.0.1:43706_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-9 (1569623047298613248)]} 0 0
   [junit4]   2> 1416789 INFO  (qtp956463895-15461) [n:127.0.0.1:57707_ 
c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:38599/collection1/&wt=javabin&version=2}{add=[1-9
 (1569623047299661824)]} 0 0
   [junit4]   2> 1416790 INFO  (qtp2062388605-15430) [n:127.0.0.1:38599_ 
c:collection1 s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= pat

[...truncated too long message...]

2> 
   [junit4]   2> 2674175 INFO  (qtp1536934259-25339) [n:127.0.0.1:40075_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={nodeName=127.0.0.1:40075_solr&onlyIfLeaderActive=true&core=testCreateCollectionAddReplica_shard1_replica_n1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
 status=400 QTime=24034
   [junit4]   2> 2675368 WARN  
(zkCallback-3905-thread-6-processing-n:127.0.0.1:37140_solr) 
[n:127.0.0.1:37140_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 2675369 INFO  (jetty-closer-3885-thread-5) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@6d9f8e19{/solr,null,UNAVAILABLE}
   [junit4]   2> 2675372 ERROR 
(SUITE-TestPolicyCloud-seed#[79F11EE92CC8BDFD]-worker) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 2675373 INFO  
(SUITE-TestPolicyCloud-seed#[79F11EE92CC8BDFD]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:63574 63574
   [junit4]   2> 2675432 INFO  (Thread-4290) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:63574 63574
   [junit4]   2> 2675436 WARN  (Thread-4290) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        13      /solr/clusterprops.json
   [junit4]   2>        13      /solr/aliases.json
   [junit4]   2>        6       /solr/configs/conf
   [junit4]   2>        5       /solr/security.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        13      /solr/clusterstate.json
   [junit4]   2>        3       
/solr/collections/testCreateCollectionSplitShard/state.json
   [junit4]   2>        3       /solr/collections/policiesTest/state.json
   [junit4]   2>        3       
/solr/overseer_elect/election/98101501513629709-127.0.0.1:40075_solr-n_0000000000
   [junit4]   2>        2       
/solr/collections/testCreateCollectionAddReplica/state.json
   [junit4]   2>        2       
/solr/overseer_elect/election/98101501513629710-127.0.0.1:65171_solr-n_0000000002
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        30      /solr/collections
   [junit4]   2>        13      /solr/live_nodes
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TestPolicyCloud_79F11EE92CC8BDFD-001
   [junit4]   2> Jun 08, 2017 8:27:23 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=831, maxMBSortInHeap=7.994612988132742, 
sim=RandomSimilarity(queryNorm=true): {}, locale=it, timezone=Cuba
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_131 
(64-bit)/cpus=3,threads=1,free=78979160,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestDistribDocBasedVersion, 
RemoteQueryErrorTest, TestInitQParser, 
TestDistributedStatsComponentCardinality, SimpleFacetsTest, DeleteStatusTest, 
TestFastOutputStream, BaseCdcrDistributedZkTest, PeerSyncReplicationTest, 
CustomCollectionTest, TestIBSimilarityFactory, 
TlogReplayBufferedWhileIndexingTest, TestMacroExpander, 
WrapperMergePolicyFactoryTest, MoveReplicaTest, 
UUIDUpdateProcessorFallbackTest, DistributedSpellCheckComponentTest, 
AlternateDirectoryTest, StandardRequestHandlerTest, 
TestSerializedLuceneMatchVersion, TestSubQueryTransformerCrossCore, 
TestConfigSetProperties, TestCloudNestedDocsSort, TestHighlightDedupGrouping, 
CloudExitableDirectoryReaderTest, TestSolrCLIRunExample, SpatialFilterTest, 
RuleEngineTest, SuggesterFSTTest, TestCollationFieldDocValues, TestUtils, 
OpenExchangeRatesOrgProviderTest, DistributedQueryElevationComponentTest, 
SolrShardReporterTest, TestLeaderInitiatedRecoveryThread, TestFieldCacheSort, 
ActionThrottleTest, TestChildDocTransformer, TestManagedSchema, 
SimpleCollectionCreateDeleteTest, PropertiesRequestHandlerTest, 
TestSolrConfigHandlerConcurrent, TestPullReplicaErrorHandling, 
RollingRestartTest, TestSolr4Spatial2, ClusterStateUpdateTest, 
TestSortByMinMaxFunction, SolrTestCaseJ4Test, SortSpecParsingTest, 
TestExportWriter, TestReplicaProperties, BufferStoreTest, 
TestWordDelimiterFilterFactory, AtomicUpdateProcessorFactoryTest, 
TestFieldCache, NumericFieldsTest, TestEmbeddedSolrServerConstructors, 
SegmentsInfoRequestHandlerTest, TestOverriddenPrefixQueryForCustomFieldType, 
CachingDirectoryFactoryTest, CdcrRequestHandlerTest, 
HdfsBasicDistributedZkTest, BooleanFieldTest, TestMergePolicyConfig, 
TestSchemaManager, HLLUtilTest, RAMDirectoryFactoryTest, TestFunctionQuery, 
DistributedMLTComponentTest, HdfsSyncSliceTest, SolrSlf4jReporterTest, 
TestObjectReleaseTracker, TestManagedResource, TestGraphTermsQParserPlugin, 
SolrCloudExampleTest, TestSlowCompositeReaderWrapper, 
TestFieldCacheVsDocValues, TestUseDocValuesAsStored, PrimitiveFieldTypeTest, 
DistributedFacetPivotSmallTest, TestSchemaSimilarityResource, 
TestComplexPhraseQParserPlugin, ZkControllerTest, FastVectorHighlighterTest, 
TestStressLiveNodes, TestConfigsApi, TestDynamicFieldCollectionResource, 
TestLockTree, TestBinaryField, TestCharFilters, TestSizeLimitedDistributedMap, 
HdfsChaosMonkeySafeLeaderTest, TestHashPartitioner, 
TestRandomRequestDistribution, SolrCoreTest, 
VMParamsZkACLAndCredentialsProvidersTest, DocumentBuilderTest, 
LeaderElectionTest, TestStressCloudBlindAtomicUpdates, InfoHandlerTest, 
MetricsConfigTest, TestLRUStatsCache, TestFieldTypeCollectionResource, 
TestSolrDeletionPolicy1, FileUtilsTest, TestCSVResponseWriter, 
ReplicationFactorTest, TestMultiValuedNumericRangeQuery, 
TestTrackingShardHandlerFactory, DateFieldTest, MBeansHandlerTest, 
SyncSliceTest, TestPhraseSuggestions, DistributedFacetPivotLongTailTest, 
TestConfigOverlay, TestCorePropertiesReload, DistributedDebugComponentTest, 
PluginInfoTest, TestStressUserVersions, TestSort, TestCoreContainer, 
OverseerModifyCollectionTest, BasicAuthStandaloneTest, TestReloadDeadlock, 
TestMinMaxOnMultiValuedField, TestShardHandlerFactory, TestSolrJ, 
TestCloudPivotFacet, ChaosMonkeyNothingIsSafeWithPullReplicasTest, 
TestManagedSynonymGraphFilterFactory, TestReRankQParserPlugin, CursorMarkTest, 
TestCollapseQParserPlugin, TestJettySolrRunner, TestLegacyField, 
ClassificationUpdateProcessorTest, SolrPluginUtilsTest, 
TestBlendedInfixSuggestions, HdfsWriteToMultipleCollectionsTest, 
TestXmlQParser, TestDynamicLoading, TestXIncludeConfig, 
TestNamedUpdateProcessors, QueryElevationComponentTest, SaslZkACLProviderTest, 
DeleteReplicaTest, UnloadDistributedZkTest, SimplePostToolTest, 
TestJsonFacetRefinement, ExitableDirectoryReaderTest, OutputWriterTest, 
CopyFieldTest, SimpleMLTQParserTest, TestJmxIntegration, 
SolrMetricsIntegrationTest, IndexSchemaRuntimeFieldTest, ResourceLoaderTest, 
OverriddenZkACLAndCredentialsProvidersTest, PeerSyncTest, 
BigEndianAscendingWordDeserializerTest, TestLazyCores, TestNRTOpen, 
TestQuerySenderNoQuery, TestCollectionAPI, HdfsNNFailoverTest, 
TestGraphMLResponseWriter, TestApiFramework, TestInfoStreamLogging, 
ClassificationUpdateProcessorFactoryTest, ShufflingReplicaListTransformerTest, 
SpellingQueryConverterTest, TestSurroundQueryParser, 
XmlUpdateRequestHandlerTest, EnumFieldTest, TestBulkSchemaConcurrent, 
TestRestoreCore, TestReplicationHandler, DirectUpdateHandlerTest, SOLR749Test, 
BasicDistributedZkTest, ShardRoutingTest, RecoveryZkTest, TestZkChroot, 
TestRandomDVFaceting, TestDistributedSearch, TestSolr4Spatial, 
BadIndexSchemaTest, TestFiltering, TestIndexSearcher, 
ShowFileRequestHandlerTest, CurrencyFieldXmlFileTest, TestCoreDiscovery, 
AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, 
TestExtendedDismaxParser, SpellCheckCollatorTest, SolrRequestParserTest, 
DocValuesTest, TestStressLucene, TestCSVLoader, PolyFieldTest, 
SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, 
TestAtomicUpdateErrorCases, QueryEqualityTest, 
StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, 
DocValuesMultiTest, DistanceFunctionTest, SolrInfoBeanTest, 
XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, 
DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, 
TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, 
FileBasedSpellCheckerTest, TermsComponentTest, 
PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, 
TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, 
FieldAnalysisRequestHandlerTest, LoggingHandlerTest, TestCollationField, 
ReturnFieldsTest, UniqFieldsUpdateProcessorFactoryTest, 
BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestLFUCache, 
SliceStateTest, UUIDFieldTest, TestRTGBase, DistributedIntervalFacetingTest, 
TestTolerantSearch, TestPathTrie, ConnectionReuseTest, 
CdcrReplicationDistributedZkTest, CollectionsAPIAsyncDistributedZkTest, 
ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, 
ConnectionManagerTest, CreateCollectionCleanupTest, DeleteShardTest, 
DistribJoinFromCollectionTest, DistributedQueueTest, 
DistributedVersionInfoTest, HttpPartitionTest, 
LeaderFailoverAfterPartitionTest, MissingSegmentRecoveryTest, NodeMutatorTest, 
OutOfBoxZkACLAndCredentialsProvidersTest, 
OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, 
RecoveryAfterSoftCommitTest, TestCloudJSONFacetJoinDomain, TestCloudRecovery, 
TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, 
TestLocalFSCloudBackupRestore, TestOnReconnectListenerSupport, 
TestRequestStatusCollectionAPI, TestSolrCloudWithKerberosAlt, 
TestTolerantUpdateProcessorCloud, TestTolerantUpdateProcessorRandomCloud, 
TestPolicyCloud]
   [junit4] Completed [597/726 (2!)] on J0 in 325.75s, 4 tests, 2 errors <<< 
FAILURES!

[...truncated 43841 lines...]

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org