Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1094/

11 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  org.apache.solr.cloud.DeleteShardTest.test

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 
127.0.0.1:44687 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: 
Could not connect to ZooKeeper 127.0.0.1:44687 within 45000 ms
        at 
__randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A:E7A8E6E34BBBB7A2]:0)
        at 
org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:181)
        at 
org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
        at 
org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:105)
        at 
org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:88)
        at 
org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:82)
        at 
org.apache.solr.cloud.AbstractDistribZkTestBase.distribSetUp(AbstractDistribZkTestBase.java:75)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribSetUp(AbstractFullDistribZkTestBase.java:197)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:934)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Could not connect to 
ZooKeeper 127.0.0.1:44687 within 45000 ms
        at 
org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:208)
        at 
org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:173)
        ... 38 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteShardTest

Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.DeleteShardTest:     
1) Thread[id=11133, name=qtp1297997705-11133, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11134, 
name=qtp370483280-11134, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11131, 
name=qtp415401506-11131, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=11070, 
name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, 
group=TGRP-DeleteShardTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=11132, 
name=qtp1868655190-11132, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    6) Thread[id=11073, 
name=ProcessThread(sid:0 cport:-1):, state=WAITING, group=TGRP-DeleteShardTest] 
        at sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
    7) Thread[id=11069, name=Thread-9987, state=WAITING, 
group=TGRP-DeleteShardTest]         at sun.misc.Unsafe.park(Native Method)      
   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
         at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:416)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)   
      at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96) 
        at 
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
         at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)   
      at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
         at 
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)    
8) Thread[id=11135, name=qtp413760287-11135, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    9) Thread[id=11071, 
name=SessionTracker, state=TIMED_WAITING, group=TGRP-DeleteShardTest]         
at java.lang.Object.wait(Native Method)         at 
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE 
scope at org.apache.solr.cloud.DeleteShardTest: 
   1) Thread[id=11133, name=qtp1297997705-11133, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11134, name=qtp370483280-11134, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11131, name=qtp415401506-11131, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=11070, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, 
state=RUNNABLE, group=TGRP-DeleteShardTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=11132, name=qtp1868655190-11132, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=11073, name=ProcessThread(sid:0 cport:-1):, state=WAITING, 
group=TGRP-DeleteShardTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
   7) Thread[id=11069, name=Thread-9987, state=WAITING, 
group=TGRP-DeleteShardTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:416)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at 
org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96)
        at 
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
        at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)
        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
        at 
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)
   8) Thread[id=11135, name=qtp413760287-11135, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=11071, name=SessionTracker, state=TIMED_WAITING, 
group=TGRP-DeleteShardTest]
        at java.lang.Object.wait(Native Method)
        at 
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteShardTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=11070, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, 
group=TGRP-DeleteShardTest]         at 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)         at 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)         at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11069, 
name=Thread-9987, state=WAITING, group=TGRP-DeleteShardTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
         at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:416)    
     at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)   
      at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96) 
        at 
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
         at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)   
      at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
         at 
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=11070, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, 
state=RUNNABLE, group=TGRP-DeleteShardTest]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11069, name=Thread-9987, state=WAITING, 
group=TGRP-DeleteShardTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:416)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
        at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
        at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at 
org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96)
        at 
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
        at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)
        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
        at 
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
        at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.QueryResultKeyTest

Error Message:


Stack Trace:
java.lang.AssertionError
        at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.QueryResultKeyTest

Error Message:
5 threads leaked from SUITE scope at org.apache.solr.core.QueryResultKeyTest:   
  1) Thread[id=10712, name=qtp1868655190-10712, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=10710, 
name=qtp370483280-10710, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=10713, 
name=qtp1297997705-10713, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=10711, 
name=qtp415401506-10711, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=10714, 
name=qtp413760287-10714, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.core.QueryResultKeyTest: 
   1) Thread[id=10712, name=qtp1868655190-10712, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=10710, name=qtp370483280-10710, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=10713, name=qtp1297997705-10713, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=10711, name=qtp415401506-10711, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=10714, name=qtp413760287-10714, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestNRTOpen

Error Message:
5 threads leaked from SUITE scope at org.apache.solr.core.TestNRTOpen:     1) 
Thread[id=11056, name=qtp415401506-11056, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=11055, 
name=qtp370483280-11055, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=11054, 
name=qtp1297997705-11054, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=11052, 
name=qtp1868655190-11052, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=11053, 
name=qtp413760287-11053, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.core.TestNRTOpen: 
   1) Thread[id=11056, name=qtp415401506-11056, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=11055, name=qtp370483280-11055, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=11054, name=qtp1297997705-11054, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=11052, name=qtp1868655190-11052, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=11053, name=qtp413760287-11053, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestQuerySenderListener

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)
        at 
org.apache.lucene.util.TestRuleRestoreSystemProperties.before(TestRuleRestoreSystemProperties.java:52)
        at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.spelling.WordBreakSolrSpellCheckerTest

Error Message:
ObjectTracker found 643 object(s) that were not released!!! 
[MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, SolrCore, MockDirectoryWrapper, SolrCore, SolrCore, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrZkClient, MockDirectoryWrapper, SolrCore, 
SolrCore, MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
SolrCore, SolrCore, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, SolrCore, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, SolrZkClient, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, RawDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrZkClient, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrZkClient, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, 
SolrCore, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrZkClient, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 643 object(s) that were not 
released!!! [MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, SolrCore, MockDirectoryWrapper, SolrCore, SolrCore, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrZkClient, MockDirectoryWrapper, SolrCore, 
SolrCore, MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
SolrCore, SolrCore, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, SolrCore, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, SolrZkClient, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, SolrCore, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, RawDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrZkClient, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MDCAwareThreadPoolExecutor, SolrCore, 
MDCAwareThreadPoolExecutor, SolrCore, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, MDCAwareThreadPoolExecutor, 
MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrZkClient, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, SolrCore, SolrCore, 
SolrCore, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, SolrZkClient, 
MockDirectoryWrapper, SolrCore, SolrCore, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor, SolrCore, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, SolrCore]
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:228)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.spelling.WordBreakSolrSpellCheckerTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.spelling.WordBreakSolrSpellCheckerTest:     1) Thread[id=10895, 
name=qtp415401506-10895, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=10897, 
name=qtp1868655190-10897, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=10898, 
name=qtp1297997705-10898, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    4) Thread[id=10899, 
name=qtp370483280-10899, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=10896, 
name=qtp413760287-10896, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)     
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.spelling.WordBreakSolrSpellCheckerTest: 
   1) Thread[id=10895, name=qtp415401506-10895, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=10897, name=qtp1868655190-10897, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=10898, name=qtp1297997705-10898, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=10899, name=qtp370483280-10899, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=10896, name=qtp413760287-10896, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)




Build Log:
[...truncated 12631 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_6FFCD939E547DA5A-001/init-core-data-001
   [junit4]   2> 135720 INFO  
(SUITE-CollectionsAPIDistributedZkTest-seed#[6FFCD939E547DA5A]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 135720 INFO  
(SUITE-CollectionsAPIDistributedZkTest-seed#[6FFCD939E547DA5A]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: 
/ii_qf/f
   [junit4]   2> 135728 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 135728 INFO  (Thread-2356) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 135728 INFO  (Thread-2356) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 135828 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:43346
   [junit4]   2> 135829 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 135829 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 135839 INFO  (zkCallback-56-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6e1cf1b5 
name:ZooKeeperConnection Watcher:127.0.0.1:43346 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135839 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 135840 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 135840 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 135845 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x152ceddf0410000, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 135846 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 135849 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 135855 INFO  (zkCallback-57-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@11902c05 
name:ZooKeeperConnection Watcher:127.0.0.1:43346/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135855 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 135856 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 135856 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 135859 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 135861 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 135863 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 135865 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 135865 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 135868 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 135869 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 135871 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 135872 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 135874 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 135874 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 135876 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 135877 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 135879 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 135879 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 135882 INFO  
(TEST-CollectionsAPIDistributedZkTest.test-seed#[6FFCD939E547DA5A]) [    ] 
o.a.s.c.AbstractZkTestCase put /x1/j

[...truncated too long message...]

bPoll(QueuedThreadPool.java:531)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    6) Thread[id=11073, name=ProcessThread(sid:0 cport:-1):, 
state=WAITING, group=TGRP-DeleteShardTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
   [junit4]    >    7) Thread[id=11069, name=Thread-9987, state=WAITING, 
group=TGRP-DeleteShardTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
   [junit4]    >         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
   [junit4]    >         at 
com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:416)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
   [junit4]    >         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
   [junit4]    >         at 
org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96)
   [junit4]    >         at 
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
   [junit4]    >         at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)
   [junit4]    >         at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
   [junit4]    >         at 
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at 
org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)
   [junit4]    >    8) Thread[id=11135, name=qtp413760287-11135, 
state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4]    >         at 
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
   [junit4]    >         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    9) Thread[id=11071, name=SessionTracker, 
state=TIMED_WAITING, group=TGRP-DeleteShardTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=11070, 
name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, 
group=TGRP-DeleteShardTest]
   [junit4]    >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native 
Method)
   [junit4]    >         at 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   [junit4]    >         at 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
   [junit4]    >         at 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   [junit4]    >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   [junit4]    >         at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=11069, name=Thread-9987, state=WAITING, 
group=TGRP-DeleteShardTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
   [junit4]    >         at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
   [junit4]    >         at 
com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:416)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
   [junit4]    >         at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
   [junit4]    >         at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
   [junit4]    >         at 
org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:96)
   [junit4]    >         at 
org.apache.zookeeper.server.ZooKeeperServer.registerJMX(ZooKeeperServer.java:377)
   [junit4]    >         at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:410)
   [junit4]    >         at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:123)
   [junit4]    >         at 
org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
   [junit4]    >         at 
org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:491)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([6FFCD939E547DA5A]:0)
   [junit4] Completed [486/568 (6!)] on J1 in 73.66s, 1 test, 3 errors <<< 
FAILURES!

[...truncated 247 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J1-20160211_054418_957.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/heapdumps/java_pid7918.hprof
 ...
   [junit4] Heap dump file created [624519490 bytes in 6.639 secs]
   [junit4] <<< JVM J1: EOF ----

   [junit4] JVM J1: stderr was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/temp/junit4-J1-20160211_054418_957.syserr
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:43346"
   [junit4] <<< JVM J1: EOF ----

[...truncated 17 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:757: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:694: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:59: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:534:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1477:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1033:
 There were test failures: 568 suites (7 ignored), 2269 tests, 9 suite-level 
errors, 2 errors, 283 ignored (10 assumptions) [seed: 6FFCD939E547DA5A]

Total time: 252 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to