[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1002 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1002/

13 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedGrouping

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedGrouping

Error Message:
Could not remove the following files (in the order of attempts):    
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002/write.lock:
 java.io.IOException: access denied: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002/write.lock
    
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002:
 java.nio.file.DirectoryNotEmptyException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002
    
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001/write.lock:
 java.io.IOException: access denied: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001/write.lock
    
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001:
 java.nio.file.DirectoryNotEmptyException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001
    
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001:
 java.nio.file.DirectoryNotEmptyException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001
 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of 
attempts):
   
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002/write.lock:
 java.io.IOException: access denied: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002/write.lock
   
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002:
 java.nio.file.DirectoryNotEmptyException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-NIOFSDirectory-002
   
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001/write.lock:
 java.io.IOException: access denied: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001/write.lock
   
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001:
 java.nio.file.DirectoryNotEmptyException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001/index-MMapDirectory-001
   
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001:
 java.nio.file.DirectoryNotEmptyException: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_15FB64166498C6A0-001

        at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323)
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedGrouping

Error Message:
62 threads leaked from SUITE scope at org.apache.solr.TestDistributedGrouping:  
   1) Thread[id=57325, 
name=coreZkRegister-4403-thread-6-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=57339, 
name=coreZkRegister-4403-thread-17-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=57292, 
name=coreCloseExecutor-6105-thread-17, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=57324, 
name=zkCallback-1500-thread-32, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=57356, 
name=zkCallback-1511-thread-28, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=57323, 
name=coreZkRegister-4403-thread-5, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    7) Thread[id=57333, 
name=coreZkRegister-4403-thread-11-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    8) Thread[id=57329, 
name=coreZkRegister-4403-thread-7-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    9) Thread[id=57340, 
name=coreZkRegister-4403-thread-18, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   10) Thread[id=57337, 
name=coreZkRegister-4403-thread-15-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   11) Thread[id=57303, 
name=zkCallback-1488-thread-17-processing-n:127.0.0.1:54629_-EventThread, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   
12) Thread[id=57334, 
name=coreZkRegister-4403-thread-12-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   13) Thread[id=57346, 
name=OverseerThreadFactory-6122-thread-2, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   14) Thread[id=57328, 
name=zkCallback-1488-thread-27, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   15) Thread[id=57283, 
name=coreCloseExecutor-6105-thread-8, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   16) Thread[id=57280, 
name=coreCloseExecutor-6105-thread-5, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   17) Thread[id=57309, 
name=zkCallback-1505-thread-24, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   18) Thread[id=57342, 
name=OverseerCollectionConfigSetProcessor-95821552135176248-127.0.0.1:47560_-n_0000000005,
 state=BLOCKED, group=Overseer collection creation process.]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at 
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1153)         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)      
   at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)      
   at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)        
 at 
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:358)
         at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:175) 
        at java.lang.Thread.run(Thread.java:745)   19) Thread[id=57321, 
name=coreZkRegister-4403-thread-3-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   20) Thread[id=57316, 
name=zkCallback-1505-thread-29, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   21) Thread[id=57331, 
name=coreZkRegister-4403-thread-9-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   22) Thread[id=57286, 
name=coreCloseExecutor-6105-thread-11, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
         at com.sun.jmx.mbeanserver.Repository.remove(Repository.java:623)      
   at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterFromRepository(DefaultMBeanServerInterceptor.java:1940)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:448)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   23) Thread[id=57284, 
name=coreCloseExecutor-6105-thread-9, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
         at com.sun.jmx.mbeanserver.Repository.remove(Repository.java:623)      
   at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterFromRepository(DefaultMBeanServerInterceptor.java:1940)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:448)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   24) Thread[id=57269, 
name=qtp1143716263-57269, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) 
        at java.lang.Thread.run(Thread.java:745)   25) Thread[id=57289, 
name=coreCloseExecutor-6105-thread-14, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   26) Thread[id=57278, 
name=coreCloseExecutor-6105-thread-3, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
         at com.sun.jmx.mbeanserver.Repository.remove(Repository.java:623)      
   at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterFromRepository(DefaultMBeanServerInterceptor.java:1940)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:448)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   27) Thread[id=57313, 
name=zkCallback-1505-thread-28, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   28) Thread[id=57282, 
name=coreCloseExecutor-6105-thread-7, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   29) Thread[id=57302, 
name=zkCallback-1488-thread-17-processing-n:127.0.0.1:54629_-SendThread(127.0.0.1:44183),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   30) 
Thread[id=57326, 
name=OverseerStateUpdate-95821552135176248-127.0.0.1:47560_-n_0000000005, 
state=WAITING, group=Overseer state updater.]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at 
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:781)         at 
org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:383)      
   at 
org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:380)      
   at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:380)         
at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:248)      
   at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.onEnqueue(Overseer.java:220)
         at 
org.apache.solr.cloud.overseer.ZkStateWriter.enqueueUpdate(ZkStateWriter.java:127)
         at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:294)
         at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:217)       
  at java.lang.Thread.run(Thread.java:745)   31) Thread[id=57281, 
name=coreCloseExecutor-6105-thread-6, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
         at com.sun.jmx.mbeanserver.Repository.remove(Repository.java:623)      
   at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterFromRepository(DefaultMBeanServerInterceptor.java:1940)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:448)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   32) Thread[id=57330, 
name=coreZkRegister-4403-thread-8, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   33) Thread[id=57308, 
name=zkCallback-1505-thread-23, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   34) Thread[id=57322, 
name=coreZkRegister-4403-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   35) Thread[id=57348, 
name=OverseerThreadFactory-6122-thread-4, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   36) Thread[id=57318, 
name=zkCallback-1511-thread-27-processing-n:127.0.0.1:47560_, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at 
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1153)         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)      
   at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)      
   at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)        
 at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1045)
         at 
org.apache.solr.common.cloud.ZkStateReader.access$500(ZkStateReader.java:58)    
     at 
org.apache.solr.common.cloud.ZkStateReader$StateWatcher.refreshAndWatch(ZkStateReader.java:912)
         at 
org.apache.solr.common.cloud.ZkStateReader$StateWatcher.process(ZkStateReader.java:897)
         at 
org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:266)
         at 
org.apache.solr.common.cloud.SolrZkClient$3$$Lambda$80/1511758408.run(Unknown 
Source)         at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         
at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   37) Thread[id=57335, 
name=coreZkRegister-4403-thread-13-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   38) Thread[id=57271, 
name=qtp1037342919-57271, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) 
        at java.lang.Thread.run(Thread.java:745)   39) Thread[id=57345, 
name=OverseerThreadFactory-6122-thread-1, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   40) Thread[id=57294, 
name=coreCloseExecutor-6105-thread-19, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   41) Thread[id=57315, 
name=zkCallback-1505-thread-25-processing-n:127.0.0.1:41155_-EventThread, 
state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   
42) Thread[id=57310, 
name=zkCallback-1505-thread-25-processing-n:127.0.0.1:41155_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1002)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:972)   
      at 
org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1559)
         at 
org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:442)
         at org.apache.solr.cloud.ZkController.access$500(ZkController.java:88) 
        at org.apache.solr.cloud.ZkController$1.command(ZkController.java:304)  
       at 
org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:163)
         at 
org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:57)
         at 
org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:137)
         at 
org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:266)
         at 
org.apache.solr.common.cloud.SolrZkClient$3$$Lambda$80/1511758408.run(Unknown 
Source)         at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         
at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   43) Thread[id=57264, 
name=qtp1143716263-57264, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
         at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) 
        at java.lang.Thread.run(Thread.java:745)   44) Thread[id=57272, 
name=qtp218596612-57272, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)
         at 
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)
         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) 
        at java.lang.Thread.run(Thread.java:745)   45) Thread[id=57349, 
name=OverseerThreadFactory-6122-thread-5, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   46) Thread[id=57277, 
name=coreCloseExecutor-6105-thread-2, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   47) Thread[id=57285, 
name=coreCloseExecutor-6105-thread-10, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   48) Thread[id=57279, 
name=coreCloseExecutor-6105-thread-4, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   49) Thread[id=57288, 
name=coreCloseExecutor-6105-thread-13, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
         at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:487)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546) 
        at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:154) 
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1210)         at 
org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:133)         at 
org.apache.solr.core.SolrCores$$Lambda$54/1700344061.call(Unknown Source)       
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   50) Thread[id=57317, 
name=zkCallback-1505-thread-30-processing-n:127.0.0.1:41155_, state=RUNNABLE, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   51) Thread[id=57312, 
name=zkCallback-1505-thread-27, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   52) Thread[id=57343, 
name=OverseerHdfsCoreFailoverThread-95821552135176248-127.0.0.1:47560_-n_0000000005,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:138)
         at java.lang.Thread.run(Thread.java:745)   53) Thread[id=57311, 
name=zkCallback-1505-thread-26, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   54) Thread[id=57320, 
name=coreZkRegister-4403-thread-2-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   55) Thread[id=57332, 
name=coreZkRegister-4403-thread-10-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   56) Thread[id=57350, 
name=OverseerThreadFactory-6122-thread-6-processing-n:127.0.0.1:47560_, 
state=WAITING, group=Overseer collection creation process.]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
         at 
org.apache.solr.handler.component.HttpShardHandler.take(HttpShardHandler.java:271)
         at 
org.apache.solr.handler.component.HttpShardHandler.takeCompletedOrError(HttpShardHandler.java:264)
         at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.processResponses(OverseerCollectionMessageHandler.java:2162)
         at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.collectionCmd(OverseerCollectionMessageHandler.java:2253)
         at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.deleteCollection(OverseerCollectionMessageHandler.java:685)
         at 
org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:220)
         at 
org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:437)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   57) Thread[id=57268, 
name=qtp214748515-57268, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) 
        at java.lang.Thread.run(Thread.java:745)   58) Thread[id=57314, 
name=zkCallback-1505-thread-25-processing-n:127.0.0.1:41155_-SendThread(127.0.0.1:44183),
 state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)   59) 
Thread[id=57319, name=zkCallback-1495-thread-34, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   60) Thread[id=57336, 
name=coreZkRegister-4403-thread-14-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   61) Thread[id=57338, 
name=coreZkRegister-4403-thread-16-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:855)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   62) Thread[id=57327, 
name=coreZkRegister-4380-thread-2, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 62 threads leaked from 
SUITE scope at org.apache.solr.TestDistributedGrouping: 
   1) Thread[id=57325, 
name=coreZkRegister-4403-thread-6-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1044)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:855)
        at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$45/1263782840.run(Unknown
 Source)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=57339, 
name=coreZkRegister-4403-thread-17-processing-n:127.0.0.1:57735_, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(Elec

[...truncated too long message...]

s/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/analysis/icu/lucene-analyzers-icu-7.0.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar
 com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J0-20160501_161129_103.events
 
@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J0-20160501_161129_103.suites
 -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from 
the forked process? This may indicate JVM crash or runner bugs.
   [junit4]     at 
com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4]     at 
com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4]     at 
com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4]     at 
com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]     at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]     at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]     at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:747:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:684:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:529:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1457:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1014:
 At least one slave process threw an exception, first: Quit event not received 
from the forked process? This may indicate JVM crash or runner bugs.

Total time: 406 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org