[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 935 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/935/

4 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.schema.SpatialRPTFieldTypeTest

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.schema.SpatialRPTFieldTypeTest

Error Message:
141 threads leaked from SUITE scope at 
org.apache.solr.schema.SpatialRPTFieldTypeTest:     1) Thread[id=15973, 
name=coreZkRegister-2188-thread-20, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=16081, 
name=qtp275353430-16081, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=15954, 
name=OverseerThreadFactory-3408-thread-4, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=16100, 
name=qtp794799659-16100, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=15965, 
name=coreZkRegister-2188-thread-12, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=15969, 
name=coreZkRegister-2188-thread-16, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    7) Thread[id=15983, 
name=coreZkRegister-2199-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    8) Thread[id=15944, 
name=coreZkRegister-2166-thread-21-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    9) Thread[id=15943, 
name=coreZkRegister-2166-thread-20-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   10) Thread[id=16160, 
name=qtp275353430-16160, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   11) Thread[id=16144, 
name=qtp794799659-16144, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   12) Thread[id=15958, 
name=OverseerThreadFactory-3408-thread-6, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   13) Thread[id=15953, 
name=coreZkRegister-2188-thread-9-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   14) Thread[id=16114, 
name=updateExecutor-1422-thread-12-processing-n:127.0.0.1:34572_ysp%2Fv 
x:awholynewstresscollection_collection0_2_shard6_replica2 s:shard6 
c:awholynewstresscollection_collection0_2 r:core_node9, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)         at 
java.util.concurrent.FutureTask.get(FutureTask.java:191)         at 
org.apache.solr.update.DefaultSolrCoreState$2.run(DefaultSolrCoreState.java:317)
         at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         
at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   15) Thread[id=16041, 
name=updateExecutor-1422-thread-9, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   16) Thread[id=15977, 
name=coreZkRegister-2199-thread-3-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   17) Thread[id=15960, 
name=coreZkRegister-2166-thread-29-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   18) Thread[id=16055, 
name=qtp275353430-16055, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   19) Thread[id=15938, 
name=coreZkRegister-2166-thread-15-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   20) Thread[id=16162, 
name=qtp715543800-16162, state=TERMINATED, group={null group}]         at 
(empty stack)   21) Thread[id=15942, name=coreZkRegister-2166-thread-19, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   22) Thread[id=15946, 
name=coreZkRegister-2166-thread-23-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   23) Thread[id=15951, 
name=coreZkRegister-2166-thread-24, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   24) Thread[id=15986, 
name=coreZkRegister-2188-thread-26, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   25) Thread[id=16053, 
name=recoveryExecutor-1429-thread-8-processing-n:127.0.0.1:33063_ysp%2Fv 
x:awholynewstresscollection_collection4_2_shard8_replica2 s:shard8 
c:awholynewstresscollection_collection4_2 r:core_node1, state=WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)         at 
java.util.concurrent.FutureTask.get(FutureTask.java:191)         at 
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:596)
         at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:353)    
     at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:224)   
      at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         
at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   26) Thread[id=16129, 
name=qtp794799659-16129, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   27) Thread[id=15998, 
name=coreZkRegister-2199-thread-10-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   28) Thread[id=15917, 
name=coreZkRegister-2166-thread-2, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   29) Thread[id=16163, 
name=qtp275353430-16163, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   30) Thread[id=15976, 
name=coreZkRegister-2199-thread-2-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   31) Thread[id=16010, 
name=coreZkRegister-2199-thread-19-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   32) Thread[id=15993, 
name=coreZkRegister-2188-thread-33-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   33) Thread[id=16124, 
name=recoveryExecutor-1429-thread-10, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   34) Thread[id=16066, 
name=qtp813813828-16066, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.core.SolrCore.closeAndWait(SolrCore.java:1020)         at 
org.apache.solr.core.CoreContainer.unload(CoreContainer.java:995)         at 
org.apache.solr.handler.admin.CoreAdminOperation$2.call(CoreAdminOperation.java:142)
         at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:676)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:439)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   35) Thread[id=15956, 
name=coreZkRegister-2166-thread-26, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   36) Thread[id=16076, 
name=qtp813813828-16076, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   37) Thread[id=16077, 
name=qtp275353430-16077, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   38) Thread[id=16015, 
name=coreZkRegister-2199-thread-24-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   39) Thread[id=15966, 
name=coreZkRegister-2188-thread-13-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   40) Thread[id=15959, 
name=coreZkRegister-2166-thread-28-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   41) Thread[id=15999, 
name=coreZkRegister-2199-thread-11-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   42) Thread[id=15916, 
name=zkCallback-1415-thread-24, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   43) Thread[id=16152, 
name=qtp794799659-16152, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   44) Thread[id=16118, 
name=qtp794799659-16118, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   45) Thread[id=15923, 
name=coreZkRegister-2188-thread-6-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   46) Thread[id=15920, 
name=coreZkRegister-2166-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   47) Thread[id=16026, 
name=recoveryExecutor-1423-thread-8, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   48) Thread[id=15991, 
name=coreZkRegister-2188-thread-31-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   49) Thread[id=15985, 
name=coreZkRegister-2188-thread-25-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   50) Thread[id=16048, 
name=recoveryExecutor-1423-thread-9-processing-n:127.0.0.1:34572_ysp%2Fv 
x:awholynewstresscollection_collection3_2_shard6_replica3 s:shard6 
c:awholynewstresscollection_collection3_2 r:core_node13, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:365)    
     at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:224)   
      at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         
at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   51) Thread[id=15929, 
name=coreZkRegister-2166-thread-6-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   52) Thread[id=16120, 
name=updateExecutor-1428-thread-10, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   53) Thread[id=15932, 
name=coreZkRegister-2166-thread-9-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   54) Thread[id=16061, 
name=qtp813813828-16061, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   55) Thread[id=16019, 
name=coreZkRegister-2199-thread-28-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   56) Thread[id=15940, 
name=coreZkRegister-2166-thread-17-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   57) Thread[id=16014, 
name=coreZkRegister-2199-thread-23, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   58) Thread[id=16013, 
name=coreZkRegister-2199-thread-22-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   59) Thread[id=15925, 
name=coreZkRegister-2166-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   60) Thread[id=15972, 
name=coreZkRegister-2188-thread-19-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at 
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)      
   at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)      
   at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)    
     at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:567)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   61) Thread[id=15980, 
name=coreZkRegister-2188-thread-23-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Object.wait(Native Method)         at 
java.lang.Object.wait(Object.java:502)         at 
org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)         at 
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1470)         at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:336)      
   at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:333)      
   at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
         at 
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:333)    
     at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:567)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   62) Thread[id=15941, 
name=coreZkRegister-2166-thread-18, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   63) Thread[id=15922, 
name=coreZkRegister-2188-thread-5-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   64) Thread[id=15921, 
name=coreZkRegister-2188-thread-4, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   65) Thread[id=16000, 
name=coreZkRegister-2199-thread-12, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   66) Thread[id=15926, 
name=OverseerThreadFactory-3408-thread-1, state=WAITING, group=Overseer 
collection creation process.]         at sun.misc.Unsafe.park(Native Method)    
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)       
  at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   67) Thread[id=16060, 
name=qtp794799659-16060, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.handler.admin.CoreAdminOperation$9.call(CoreAdminOperation.java:568)
         at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354)
         at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:676)  
       at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:439)      
   at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)    
     at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)     
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)      
   at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)       
  at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)   
      at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
       at org.eclipse.jetty.server.Server.handle(Server.java:499)         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)     
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)      
   at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
        at java.lang.Thread.run(Thread.java:745)   68) Thread[id=16018, 
name=coreZkRegister-2199-thread-27-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   69) Thread[id=16108, 
name=qtp813813828-16108, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   70) Thread[id=16007, 
name=coreZkRegister-2199-thread-16-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   71) Thread[id=15974, 
name=coreZkRegister-2188-thread-21-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   72) Thread[id=15963, 
name=coreZkRegister-2188-thread-10-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   73) Thread[id=15962, 
name=coreZkRegister-2166-thread-31-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   74) Thread[id=15978, 
name=zkCallback-1431-thread-28, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   75) Thread[id=15987, 
name=coreZkRegister-2188-thread-27, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   76) Thread[id=16063, 
name=zkCallback-1431-thread-30, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   77) Thread[id=16101, 
name=qtp794799659-16101, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   78) Thread[id=15931, 
name=coreZkRegister-2166-thread-8-processing-n:127.0.0.1:57239_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   79) Thread[id=16149, 
name=qtp794799659-16149, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)   80) Thread[id=16009, 
name=coreZkRegister-2199-thread-18-processing-n:127.0.0.1:33063_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   81) Thread[id=15961, 
name=coreZkRegister-2166-thread-30, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   82) Thread[id=15981, 
name=coreZkRegister-2188-thread-24-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:604)
         at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:306)
         at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197) 
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)    
     at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)        
 at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1060)     
    at org.apache.solr.cloud.ZkController.register(ZkController.java:871)       
  at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)   83) Thread[id=15990, 
name=coreZkRegister-2188-thread-30-processing-n:127.0.0.1:34572_ysp%2Fv, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1018)       
  at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:988)   
      at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:944)    
     at org.apache.solr.cloud.ZkController.register(ZkController.java:883)      
   at 
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
         at java.util.concurrent

[...truncated too long message...]

est]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   105) Thread[id=15971, name=coreZkRegister-2188-thread-18, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   106) Thread[id=15978, name=zkCallback-1431-thread-28, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   107) Thread[id=15987, name=coreZkRegister-2188-thread-27, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   108) Thread[id=15945, name=coreZkRegister-2166-thread-22, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   109) Thread[id=16063, name=zkCallback-1431-thread-30, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   110) Thread[id=15931, name=coreZkRegister-2166-thread-8, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([AED446C00D3545C1]:0)
   [junit4] Completed [536/579 (3!)] on J1 in 77.18s, 0 tests, 1 failure, 2 
errors <<< FAILURES!

[...truncated 129 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20160215_191322_630.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid28059.hprof
 ...
   [junit4] Heap dump file created [666880762 bytes in 5.874 secs]
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20160215_191322_630.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "collection1"
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "collection3"
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "collection2"
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "collection4"
   [junit4] <<< JVM J1: EOF ----

[...truncated 12 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:747:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:684:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:524:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1457:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1014:
 There were test failures: 579 suites (8 ignored), 2321 tests, 4 suite-level 
errors, 1 failure, 161 ignored (17 assumptions) [seed: AED446C00D3545C1]

Total time: 434 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org