Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/853/

5 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [SolrZkClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [SolrZkClient]
        at __randomizedtesting.SeedInfo.seed([31D5840E093BAE2]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:237)
        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
6 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=71215, 
name=zkCallback-3213-thread-1, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=71213, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-EventThread, 
state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    
3) Thread[id=71236, name=zkCallback-3213-thread-2, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=71238, 
name=zkCallback-3213-thread-4, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=71212, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-SendThread(127.0.0.1:38032),
 state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    
6) Thread[id=71237, name=zkCallback-3213-thread-3, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 6 threads leaked from SUITE 
scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=71215, name=zkCallback-3213-thread-1, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=71213, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-EventThread, 
state=WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   3) Thread[id=71236, name=zkCallback-3213-thread-2, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=71238, name=zkCallback-3213-thread-4, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=71212, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-SendThread(127.0.0.1:38032),
 state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   6) Thread[id=71237, name=zkCallback-3213-thread-3, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([31D5840E093BAE2]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=71215, name=zkCallback-3213-thread-1, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=71236, 
name=zkCallback-3213-thread-2, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=71238, 
name=zkCallback-3213-thread-4, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=71212, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-SendThread(127.0.0.1:38032),
 state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    5) 
Thread[id=71237, name=zkCallback-3213-thread-3, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=71215, name=zkCallback-3213-thread-1, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=71236, name=zkCallback-3213-thread-2, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=71238, name=zkCallback-3213-thread-4, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=71212, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-SendThread(127.0.0.1:38032),
 state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   5) Thread[id=71237, name=zkCallback-3213-thread-3, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([31D5840E093BAE2]:0)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=3408, name=collection4, 
state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=3408, name=collection4, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:42113/ns_jbn/f, 
http://127.0.0.1:34633/ns_jbn/f, http://127.0.0.1:50708/ns_jbn/f, 
http://127.0.0.1:33288/ns_jbn/f, http://127.0.0.1:59421/ns_jbn/f]
        at __randomizedtesting.SeedInfo.seed([31D5840E093BAE2]:0)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:886)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live 
SolrServers available to handle this request:[http://127.0.0.1:42113/ns_jbn/f, 
http://127.0.0.1:34633/ns_jbn/f, http://127.0.0.1:50708/ns_jbn/f, 
http://127.0.0.1:33288/ns_jbn/f, http://127.0.0.1:59421/ns_jbn/f]
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1099)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:883)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:59421/ns_jbn/f: KeeperErrorCode = Session 
expired for /overseer/collection-queue-work/qnr-
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
        ... 5 more


FAILED:  
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
        at 
__randomizedtesting.SeedInfo.seed([31D5840E093BAE2:DD97363DDB31D811]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:170)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:830)
        at 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR(LeaderInitiatedRecoveryOnShardRestartTest.java:109)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9838 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/init-core-data-001
   [junit4]   2> 301912 INFO  
(SUITE-CollectionsAPIDistributedZkTest-seed#[31D5840E093BAE2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 301912 INFO  
(SUITE-CollectionsAPIDistributedZkTest-seed#[31D5840E093BAE2]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: 
/ns_jbn/f
   [junit4]   2> 301915 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 301915 INFO  (Thread-1496) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 301915 INFO  (Thread-1496) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 302015 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:56469
   [junit4]   2> 302015 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 302017 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 302019 INFO  (zkCallback-182-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5629749d 
name:ZooKeeperConnection Watcher:127.0.0.1:56469 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302019 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 302019 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 302019 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 302027 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 302027 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 302030 INFO  (zkCallback-183-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4e560d6f 
name:ZooKeeperConnection Watcher:127.0.0.1:56469/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302030 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 302031 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 302031 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 302033 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 302034 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 302035 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 302037 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 302037 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 302040 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 302040 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 302042 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 302042 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 302043 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 302044 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 302045 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 302045 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 302047 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 302047 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 302049 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 302049 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 302050 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 302051 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 302052 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 302052 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 302054 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 302054 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 302056 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 302056 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 302058 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 302066 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 302068 INFO  (zkCallback-184-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@13d77b3e 
name:ZooKeeperConnection Watcher:127.0.0.1:56469 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302068 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 302068 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 302068 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 302070 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 302075 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 302079 INFO  (zkCallback-185-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@681a79fc 
name:ZooKeeperConnection Watcher:127.0.0.1:56469/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302079 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 302079 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 302079 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf2/solrconfig.xml
   [junit4]   2> 302080 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 302081 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf2/schema.xml
   [junit4]   2> 302082 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 302083 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf2/enumsConfig.xml
   [junit4]   2> 302083 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 302084 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 302085 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 302086 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf2/stopwords.txt
   [junit4]   2> 302086 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 302087 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf2/protwords.txt
   [junit4]   2> 302087 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 302088 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf2/currency.xml
   [junit4]   2> 302089 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 302090 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 302090 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 302091 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 302091 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 302096 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf2/old_synonyms.txt
   [junit4]   2> 302096 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 302098 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf2/synonyms.txt
   [junit4]   2> 302098 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 302100 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/elevate.xml
 to /configs/conf2/elevate.xml
   [junit4]   2> 302100 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 302417 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1
   [junit4]   2> 302421 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 302431 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6b005df9{/ns_jbn/f,null,AVAILABLE}
   [junit4]   2> 302434 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@6c0a9645{HTTP/1.1}{127.0.0.1:33288}
   [junit4]   2> 302434 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.e.j.s.Server Started @305365ms
   [junit4]   2> 302435 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/tempDir-001/control/data,
 hostContext=/ns_jbn/f, hostPort=33288, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores}
   [junit4]   2> 302435 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 302435 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/'
   [junit4]   2> 302435 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 302435 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 302452 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 302452 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 302453 INFO  (zkCallback-186-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3b26bd92 
name:ZooKeeperConnection Watcher:127.0.0.1:56469/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302454 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 302454 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 302455 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 302455 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/solr.xml
   [junit4]   2> 302460 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores
   [junit4]   2> 302460 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 452005108
   [junit4]   2> 302460 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/]
   [junit4]   2> 302460 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/lib
   [junit4]   2> 302461 WARN  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/lib).
   [junit4]   2> 302468 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 302469 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 302469 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 302469 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 302469 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56469/solr
   [junit4]   2> 302470 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 302470 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 302471 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 302472 INFO  (zkCallback-188-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5720d069 
name:ZooKeeperConnection Watcher:127.0.0.1:56469 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302472 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 302473 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 302479 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ConnectionManager Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 302486 INFO  
(zkCallback-189-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@50a76dde 
name:ZooKeeperConnection Watcher:127.0.0.1:56469/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 302486 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ConnectionManager Client is 
connected to ZooKeeper
   [junit4]   2> 302487 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue
   [junit4]   2> 302489 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 302492 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 302495 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 302497 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 302499 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 302500 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/aliases.json
   [junit4]   2> 302501 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/clusterstate.json
   [junit4]   2> 302503 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/security.json
   [junit4]   2> 302503 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.ZkController Register node as live 
in ZooKeeper:/live_nodes/127.0.0.1:33288_ns_jbn%2Ff
   [junit4]   2> 302504 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:33288_ns_jbn%2Ff
   [junit4]   2> 302505 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect
   [junit4]   2> 302507 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 302508 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 302508 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.LeaderElector Joined leadership 
election with path: 
/overseer_elect/election/94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000
   [junit4]   2> 302509 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.OverseerElectionContext I am going 
to be the leader 127.0.0.1:33288_ns_jbn%2Ff
   [junit4]   2> 302509 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 302510 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.Overseer Overseer 
(id=94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) starting
   [junit4]   2> 302512 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue-work
   [junit4]   2> 302517 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.OverseerAutoReplicaFailoverThread 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=10000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 302518 INFO  
(OverseerCollectionConfigSetProcessor-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000)
 [n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.OverseerTaskProcessor Process 
current queue of overseer operations
   [junit4]   2> 302518 INFO  
(OverseerStateUpdate-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.Overseer Starting to work on the 
main queue
   [junit4]   2> 302519 INFO  
(OverseerStateUpdate-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ZkStateReader Updating cluster 
state from ZooKeeper... 
   [junit4]   2> 302523 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ZkStateReader Updating cluster 
state from ZooKeeper... 
   [junit4]   2> 302537 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.CoreContainer Security conf doesn't 
exist. Skipping setup for authorization module.
   [junit4]   2> 302537 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.CoreContainer No authentication 
plugin used.
   [junit4]   2> 302538 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.CoresLocator Looking for core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores
   [junit4]   2> 302539 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1,
 collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 302539 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.CoresLocator Found core collection1 
in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/
   [junit4]   2> 302539 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 302540 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 302540 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[31D5840E093BAE2]) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 302540 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 302540 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 302542 INFO  
(OverseerStateUpdate-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.Overseer processMessage: queueSize: 
1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33288/ns_jbn/f";,
   [junit4]   2>   "node_name":"127.0.0.1:33288_ns_jbn%2Ff",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 302542 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 302542 INFO  
(OverseerStateUpdate-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33288/ns_jbn/f";,
   [junit4]   2>   "node_name":"127.0.0.1:33288_ns_jbn%2Ff",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 302543 INFO  
(OverseerStateUpdate-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.o.ClusterStateMutator building a new 
cName: control_collection
   [junit4]   2> 302543 INFO  
(OverseerStateUpdate-94869123122462726-127.0.0.1:33288_ns_jbn%2Ff-n_0000000000) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.o.ReplicaMutator Assigning new node 
to shard shard=shard1
   [junit4]   2> 302546 INFO  
(zkCallback-189-thread-2-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 302547 INFO  
(zkCallback-189-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff    ] o.a.s.c.c.ZkStateReader A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 303543 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 303543 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 303543 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 303543 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 303544 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 303544 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/'
   [junit4]   2> 303544 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 303544 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 303558 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 303563 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 303567 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 303585 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.Config Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 303586 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 303591 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 303675 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 303683 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 303684 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 303702 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 303704 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 303706 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 303707 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 303707 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 303707 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 303708 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 303708 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 303709 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 303709 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 303709 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 303709 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 303710 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr 
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@d21cd44
   [junit4]   2> 303710 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/data
   [junit4]   2> 303710 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/data/index/
   [junit4]   2> 303710 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 303711 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/data/index
   [junit4]   2> 303711 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=48, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=90.1142578125, 
floorSegmentMB=0.3701171875, forceMergeDeletesPctAllowed=27.898467809230038, 
segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.2933434028564743
   [junit4]   2> 303711 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        commit{dir=RawDirectoryWrapper(RAMDirectory@2eb3dd7e 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@33f3c1),segFN=segments_1,generation=1}
   [junit4]   2> 303711 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 303712 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup 
for core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/data/
   [junit4]   2> 303712 WARN  
(OldIndexDirectoryCleanupThreadForCore-collection1) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.DirectoryFactory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_31D5840E093BAE2-001/control-001/cores/collection1/data/
 does not point to a valid data directory; skipping clean-up of old index 
directories.
   [junit4]   2> 303716 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 303716 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 303716 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 303717 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 303717 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 303717 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 303717 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 303717 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 303717 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as 
default, creating implicit default
   [junit4]   2> 303718 WARN  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler 
{type = requestHandler,name = /dump,class = DumpRequestHandler,args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 303719 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 303720 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 303721 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 303722 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 303726 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/dump,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 303726 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 303727 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 303727 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
numVersionBuckets=65536
   [junit4]   2> 303728 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 303728 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 303729 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.5839907559182581]
   [junit4]   2> 303729 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        commit{dir=RawDirectoryWrapper(RAMDirectory@2eb3dd7e 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@33f3c1),segFN=segments_1,generation=1}
   [junit4]   2> 303729 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 303729 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@4e52797c[collection1] 
main
   [junit4]   2> 303729 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 303730 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection 
configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 303730 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based 
storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 303731 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO 
with znodeBase: /configs/conf1
   [junit4]   2> 303731 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 303731 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 303733 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 303733 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 303739 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 303740 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 303740 INFO  
(searcherExecutor-646-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff 
x:collection1 s:shard1 c:control_collection r:core_node1) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher 
Searcher@4e52797c[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 303740 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed 
version buckets
   [junit4]   2> 303740 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 
65536 version buckets from index
   [junit4]   2> 303740 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed 
version bucket highest value from index
   [junit4]   2> 303740 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent 
updates, using new clock 1517905971872006144
   [junit4]   2> 303743 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with 
highest version 1517905971872006144
   [junit4]   2> 303743 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 303743 INFO  
(coreLoadExecutor-645-thread-1-processing-n:127.0.0.1:33288_ns_jbn%2Ff) 
[n:127.0.0.1:33288_ns_jbn%2Ff c

[...truncated too long message...]

   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >    3) Thread[id=71236, name=zkCallback-3213-thread-2, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=71238, name=zkCallback-3213-thread-4, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=71212, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-SendThread(127.0.0.1:38032),
 state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    6) Thread[id=71237, name=zkCallback-3213-thread-3, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([31D5840E093BAE2]:0)Throwable #3: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=71215, name=zkCallback-3213-thread-1, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=71236, name=zkCallback-3213-thread-2, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=71238, name=zkCallback-3213-thread-4, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    4) Thread[id=71212, 
name=zkCallback-3197-thread-3-processing-n:127.0.0.1:34793_n%2Fhk-SendThread(127.0.0.1:38032),
 state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    5) Thread[id=71237, name=zkCallback-3213-thread-3, 
state=TIMED_WAITING, group=TGRP-CdcrReplicationDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([31D5840E093BAE2]:0)
   [junit4] Completed [443/558] on J0 in 967.14s, 11 tests, 1 failure, 2 errors 
<<< FAILURES!

[...truncated 359 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 558 suites (8 ignored), 2214 tests, 3 suite-level 
errors, 1 error, 1 failure, 428 ignored (18 assumptions) [seed: 31D5840E093BAE2]

Total time: 215 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to