[JENKINS] Lucene-Solr-NightlyTests-master - Build # 994 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/994/

19 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Captured an uncaught exception in thread: Thread[id=64457, name=Thread-55643, 
state=RUNNABLE, group=TGRP-TestDistributedSearch]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=64457, name=Thread-55643, state=RUNNABLE, 
group=TGRP-TestDistributedSearch]
        at 
__randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A:933668B755A1CFB2]:0)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:45281//collection1: 
java.lang.NullPointerException
        at 
org.apache.solr.search.grouping.distributed.responseprocessor.TopGroupsShardResponseProcessor.process(TopGroupsShardResponseProcessor.java:105)
        at 
org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:753)
        at 
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:736)
        at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:420)
        at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:2015)
        at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:518)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
        at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
        at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
        at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
        at java.lang.Thread.run(Thread.java:745)

        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:606)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
        at 
org.apache.solr.TestDistributedSearch$1.run(TestDistributedSearch.java:1124)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
ObjectTracker found 12 object(s) that were not released!!! [InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 12 object(s) that were not 
released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient]
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:255)
        at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationHandlerTest

Error Message:
12 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrReplicationHandlerTest:     1) Thread[id=106570, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=106107, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=106690, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=106691, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=105580, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=106934, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=105579, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=106108, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=105843, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=105844, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=106571, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=106933, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]         at java.lang.Thread.sleep(Native 
Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from 
SUITE scope at org.apache.solr.cloud.CdcrReplicationHandlerTest: 
   1) Thread[id=106570, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=106107, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=106690, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=106691, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=105580, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=106934, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=105579, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=106108, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   9) Thread[id=105843, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  10) Thread[id=105844, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  11) Thread[id=106571, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
  12) Thread[id=106933, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationHandlerTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:34199

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:34199
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:617)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:458)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
        ... 47 more


FAILED:  org.apache.solr.cloud.CustomCollectionTest.test

Error Message:
Error from server at http://127.0.0.1:39818: KeeperErrorCode = NoNode for 
/overseer/collection-queue-work/qnr-0000000008

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:39818: KeeperErrorCode = NoNode for 
/overseer/collection-queue-work/qnr-0000000008
        at 
__randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A:933668B755A1CFB2]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:606)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:246)
        at 
org.apache.solr.cloud.CustomCollectionTest.test(CustomCollectionTest.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestConfigSetsAPI.testDeleteErrors

Error Message:
Captured an uncaught exception in thread: Thread[id=23639, 
name=OverseerThreadFactory-6598-thread-1, state=RUNNABLE, group=Overseer 
collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=23639, name=OverseerThreadFactory-6598-thread-1, 
state=RUNNABLE, group=Overseer collection creation process.]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
        at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:70)
        at 
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:386)
        at 
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:112)
        at 
sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:340)
        at 
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
        at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at java.util.ArrayList.writeObject(ArrayList.java:762)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)


FAILED:  org.apache.solr.cloud.TestConfigSetsAPI.testCreateErrors

Error Message:
Captured an uncaught exception in thread: Thread[id=23798, 
name=OverseerThreadFactory-6622-thread-1, state=RUNNABLE, group=Overseer 
collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=23798, name=OverseerThreadFactory-6622-thread-1, 
state=RUNNABLE, group=Overseer collection creation process.]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
        at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:70)
        at 
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:386)
        at 
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:112)
        at 
sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:340)
        at 
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
        at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at java.util.ArrayList.writeObject(ArrayList.java:762)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestConfigSetsAPI

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestConfigSetsAPI:   
  1) Thread[id=23793, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestConfigSetsAPI]         at java.lang.Thread.sleep(Native Method)  
       at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=23634, 
name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestConfigSetsAPI]     
    at java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE 
scope at org.apache.solr.cloud.TestConfigSetsAPI: 
   1) Thread[id=23793, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestConfigSetsAPI]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=23634, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestConfigSetsAPI]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)


FAILED:  org.apache.solr.cloud.TestConfigSetsAPIExclusivity.testAPIExclusivity

Error Message:
Captured an uncaught exception in thread: Thread[id=28988, 
name=OverseerThreadFactory-8216-thread-4, state=RUNNABLE, group=Overseer 
collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=28988, name=OverseerThreadFactory-8216-thread-4, 
state=RUNNABLE, group=Overseer collection creation process.]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
        at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:70)
        at 
sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:386)
        at 
sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:112)
        at 
sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:340)
        at 
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
        at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at java.util.ArrayList.writeObject(ArrayList.java:762)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:43426/x_/sr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:43426/x_/sr
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:617)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:516)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
        at 
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
        at 
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
        at 
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
        ... 47 more


FAILED:  
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
        at 
__randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A:4D8269A2B3D648F]:0)
        at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
        at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
        at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh(ZkStateReaderTest.java:41)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Sat Apr 23 07:43:41 
EDT 2016

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to 
exceed: Sat Apr 23 07:43:41 EDT 2016
        at 
__randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A:C0C957ABFE75CBF9]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1421)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:773)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1461412769245,generation=2,filelist=[_go1.fdt, 
_go1.fdx, _go1.fnm, _go1.nvd, _go1.nvm, _go1.si, _go1_SimpleText_0.pst, 
_h3l.cfe, _h3l.cfs, _h3l.si, _h3m.cfe, _h3m.cfs, _h3m.si, _h3n.cfe, _h3n.cfs, 
_h3n.si, _h3o.cfe, _h3o.cfs, _h3o.si, _h3p.cfe, _h3p.cfs, _h3p.si, _h3q.cfe, 
_h3q.cfs, _h3q.si, _h3r.cfe, _h3r.cfs, _h3r.si, _h3s.cfe, _h3s.cfs, _h3s.si, 
_h3t.cfe, _h3t.cfs, _h3t.si, _h3u.cfe, _h3u.cfs, _h3u.si, _h3v.cfe, _h3v.cfs, 
_h3v.si, _h3w.cfe, _h3w.cfs, _h3w.si, _h3x.cfe, _h3x.cfs, _h3x.si, _h3y.cfe, 
_h3y.cfs, _h3y.si, _h3z.cfe, _h3z.cfs, _h3z.si, _h40.cfe, _h40.cfs, _h40.si, 
_h41.cfe, _h41.cfs, _h41.si, _h42.cfe, _h42.cfs, _h42.si, _h43.cfe, _h43.cfs, 
_h43.si, _h44.cfe, _h44.cfs, _h44.si, _h45.cfe, _h45.cfs, _h45.si, _h46.cfe, 
_h46.cfs, _h46.si, _h47.cfe, _h47.cfs, _h47.si, _h48.cfe, _h48.cfs, _h48.si, 
_h49.cfe, _h49.cfs, _h49.si, _h4a.cfe, _h4a.cfs, _h4a.si, _h4b.cfe, _h4b.cfs, 
_h4b.si, _h4c.cfe, _h4c.cfs, _h4c.si, _h4d.cfe, _h4d.cfs, _h4d.si, _h4e.cfe, 
_h4e.cfs, _h4e.si, _h4f.cfe, _h4f.cfs, _h4f.si, _h4g.cfe, _h4g.cfs, _h4g.si, 
_h4h.cfe, _h4h.cfs, _h4h.si, _h4i.cfe, _h4i.cfs, _h4i.si, _h4j.cfe, _h4j.cfs, 
_h4j.si, _h4k.cfe, _h4k.cfs, _h4k.si, _h4l.cfe, _h4l.cfs, _h4l.si, _h4m.cfe, 
_h4m.cfs, _h4m.si, _h4n.cfe, _h4n.cfs, _h4n.si, _hk9.cfe, _hk9.cfs, _hk9.si, 
_hka.cfe, _hka.cfs, _hka.si, _hkb.cfe, _hkb.cfs, _hkb.si, _hkc.cfe, _hkc.cfs, 
_hkc.si, _hkd.cfe, _hkd.cfs, _hkd.si, _hke.cfe, _hke.cfs, _hke.si, _hkf.cfe, 
_hkf.cfs, _hkf.si, _hkg.cfe, _hkg.cfs, _hkg.si, _hkh.cfe, _hkh.cfs, _hkh.si, 
_hki.cfe, _hki.cfs, _hki.si, _hkj.cfe, _hkj.cfs, _hkj.si, _hkk.cfe, _hkk.cfs, 
_hkk.si, _hkl.cfe, _hkl.cfs, _hkl.si, _hkm.cfe, _hkm.cfs, _hkm.si, _hkn.cfe, 
_hkn.cfs, _hkn.si, _hko.cfe, _hko.cfs, _hko.si, _hkp.cfe, _hkp.cfs, _hkp.si, 
_hkq.cfe, _hkq.cfs, _hkq.si, _hkr.cfe, _hkr.cfs, _hkr.si, _hks.cfe, _hks.cfs, 
_hks.si, _hkt.cfe, _hkt.cfs, _hkt.si, _hku.cfe, _hku.cfs, _hku.si, _hkv.cfe, 
_hkv.cfs, _hkv.si, _hkw.cfe, _hkw.cfs, _hkw.si, _hkx.cfe, _hkx.cfs, _hkx.si, 
_hky.cfe, _hky.cfs, _hky.si, _hkz.cfe, _hkz.cfs, _hkz.si, _hl0.cfe, _hl0.cfs, 
_hl0.si, _hl1.cfe, _hl1.cfs, _hl1.si, _hl2.cfe, _hl2.cfs, _hl2.si, _hl3.cfe, 
_hl3.cfs, _hl3.si, segments_2]}]> but 
was:<[{indexVersion=1461412769245,generation=2,filelist=[_go1.fdt, _go1.fdx, 
_go1.fnm, _go1.nvd, _go1.nvm, _go1.si, _go1_SimpleText_0.pst, _h3l.cfe, 
_h3l.cfs, _h3l.si, _h3m.cfe, _h3m.cfs, _h3m.si, _h3n.cfe, _h3n.cfs, _h3n.si, 
_h3o.cfe, _h3o.cfs, _h3o.si, _h3p.cfe, _h3p.cfs, _h3p.si, _h3q.cfe, _h3q.cfs, 
_h3q.si, _h3r.cfe, _h3r.cfs, _h3r.si, _h3s.cfe, _h3s.cfs, _h3s.si, _h3t.cfe, 
_h3t.cfs, _h3t.si, _h3u.cfe, _h3u.cfs, _h3u.si, _h3v.cfe, _h3v.cfs, _h3v.si, 
_h3w.cfe, _h3w.cfs, _h3w.si, _h3x.cfe, _h3x.cfs, _h3x.si, _h3y.cfe, _h3y.cfs, 
_h3y.si, _h3z.cfe, _h3z.cfs, _h3z.si, _h40.cfe, _h40.cfs, _h40.si, _h41.cfe, 
_h41.cfs, _h41.si, _h42.cfe, _h42.cfs, _h42.si, _h43.cfe, _h43.cfs, _h43.si, 
_h44.cfe, _h44.cfs, _h44.si, _h45.cfe, _h45.cfs, _h45.si, _h46.cfe, _h46.cfs, 
_h46.si, _h47.cfe, _h47.cfs, _h47.si, _h48.cfe, _h48.cfs, _h48.si, _h49.cfe, 
_h49.cfs, _h49.si, _h4a.cfe, _h4a.cfs, _h4a.si, _h4b.cfe, _h4b.cfs, _h4b.si, 
_h4c.cfe, _h4c.cfs, _h4c.si, _h4d.cfe, _h4d.cfs, _h4d.si, _h4e.cfe, _h4e.cfs, 
_h4e.si, _h4f.cfe, _h4f.cfs, _h4f.si, _h4g.cfe, _h4g.cfs, _h4g.si, _h4h.cfe, 
_h4h.cfs, _h4h.si, _h4i.cfe, _h4i.cfs, _h4i.si, _h4j.cfe, _h4j.cfs, _h4j.si, 
_h4k.cfe, _h4k.cfs, _h4k.si, _h4l.cfe, _h4l.cfs, _h4l.si, _h4m.cfe, _h4m.cfs, 
_h4m.si, _h4n.cfe, _h4n.cfs, _h4n.si, _hk9.cfe, _hk9.cfs, _hk9.si, _hka.cfe, 
_hka.cfs, _hka.si, _hkb.cfe, _hkb.cfs, _hkb.si, _hkc.cfe, _hkc.cfs, _hkc.si, 
_hkd.cfe, _hkd.cfs, _hkd.si, _hke.cfe, _hke.cfs, _hke.si, _hkf.cfe, _hkf.cfs, 
_hkf.si, _hkg.cfe, _hkg.cfs, _hkg.si, _hkh.cfe, _hkh.cfs, _hkh.si, _hki.cfe, 
_hki.cfs, _hki.si, _hkj.cfe, _hkj.cfs, _hkj.si, _hkk.cfe, _hkk.cfs, _hkk.si, 
_hkl.cfe, _hkl.cfs, _hkl.si, _hkm.cfe, _hkm.cfs, _hkm.si, _hkn.cfe, _hkn.cfs, 
_hkn.si, _hko.cfe, _hko.cfs, _hko.si, _hkp.cfe, _hkp.cfs, _hkp.si, _hkq.cfe, 
_hkq.cfs, _hkq.si, _hkr.cfe, _hkr.cfs, _hkr.si, _hks.cfe, _hks.cfs, _hks.si, 
_hkt.cfe, _hkt.cfs, _hkt.si, _hku.cfe, _hku.cfs, _hku.si, _hkv.cfe, _hkv.cfs, 
_hkv.si, _hkw.cfe, _hkw.cfs, _hkw.si, _hkx.cfe, _hkx.cfs, _hkx.si, _hky.cfe, 
_hky.cfs, _hky.si, _hkz.cfe, _hkz.cfs, _hkz.si, _hl0.cfe, _hl0.cfs, _hl0.si, 
_hl1.cfe, _hl1.cfs, _hl1.si, _hl2.cfe, _hl2.cfs, _hl2.si, _hl3.cfe, _hl3.cfs, 
_hl3.si, segments_2]}, 
{indexVersion=1461412769245,generation=3,filelist=[_h4q.fdt, _h4q.fdx, 
_h4q.fnm, _h4q.nvd, _h4q.nvm, _h4q.si, _h4q_SimpleText_0.pst, _hk9.cfe, 
_hk9.cfs, _hk9.si, _hka.cfe, _hka.cfs, _hka.si, _hkb.cfe, _hkb.cfs, _hkb.si, 
_hkc.cfe, _hkc.cfs, _hkc.si, _hkd.cfe, _hkd.cfs, _hkd.si, _hke.cfe, _hke.cfs, 
_hke.si, _hkf.cfe, _hkf.cfs, _hkf.si, _hkg.cfe, _hkg.cfs, _hkg.si, _hkh.cfe, 
_hkh.cfs, _hkh.si, _hki.cfe, _hki.cfs, _hki.si, _hkj.cfe, _hkj.cfs, _hkj.si, 
_hkk.cfe, _hkk.cfs, _hkk.si, _hkl.cfe, _hkl.cfs, _hkl.si, _hkm.cfe, _hkm.cfs, 
_hkm.si, _hkn.cfe, _hkn.cfs, _hkn.si, _hko.cfe, _hko.cfs, _hko.si, _hkp.cfe, 
_hkp.cfs, _hkp.si, _hkq.cfe, _hkq.cfs, _hkq.si, _hkr.cfe, _hkr.cfs, _hkr.si, 
_hks.cfe, _hks.cfs, _hks.si, _hkt.cfe, _hkt.cfs, _hkt.si, _hku.cfe, _hku.cfs, 
_hku.si, _hkv.cfe, _hkv.cfs, _hkv.si, _hkw.cfe, _hkw.cfs, _hkw.si, _hkx.cfe, 
_hkx.cfs, _hkx.si, _hky.cfe, _hky.cfs, _hky.si, _hkz.cfe, _hkz.cfs, _hkz.si, 
_hl0.cfe, _hl0.cfs, _hl0.si, _hl1.cfe, _hl1.cfs, _hl1.si, _hl2.cfe, _hl2.cfs, 
_hl2.si, _hl3.cfe, _hl3.cfs, _hl3.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: 
expected:<[{indexVersion=1461412769245,generation=2,filelist=[_go1.fdt, 
_go1.fdx, _go1.fnm, _go1.nvd, _go1.nvm, _go1.si, _go1_SimpleText_0.pst, 
_h3l.cfe, _h3l.cfs, _h3l.si, _h3m.cfe, _h3m.cfs, _h3m.si, _h3n.cfe, _h3n.cfs, 
_h3n.si, _h3o.cfe, _h3o.cfs, _h3o.si, _h3p.cfe, _h3p.cfs, _h3p.si, _h3q.cfe, 
_h3q.cfs, _h3q.si, _h3r.cfe, _h3r.cfs, _h3r.si, _h3s.cfe, _h3s.cfs, _h3s.si, 
_h3t.cfe, _h3t.cfs, _h3t.si, _h3u.cfe, _h3u.cfs, _h3u.si, _h3v.cfe, _h3v.cfs, 
_h3v.si, _h3w.cfe, _h3w.cfs, _h3w.si, _h3x.cfe, _h3x.cfs, _h3x.si, _h3y.cfe, 
_h3y.cfs, _h3y.si, _h3z.cfe, _h3z.cfs, _h3z.si, _h40.cfe, _h40.cfs, _h40.si, 
_h41.cfe, _h41.cfs, _h41.si, _h42.cfe, _h42.cfs, _h42.si, _h43.cfe, _h43.cfs, 
_h43.si, _h44.cfe, _h44.cfs, _h44.si, _h45.cfe, _h45.cfs, _h45.si, _h46.cfe, 
_h46.cfs, _h46.si, _h47.cfe, _h47.cfs, _h47.si, _h48.cfe, _h48.cfs, _h48.si, 
_h49.cfe, _h49.cfs, _h49.si, _h4a.cfe, _h4a.cfs, _h4a.si, _h4b.cfe, _h4b.cfs, 
_h4b.si, _h4c.cfe, _h4c.cfs, _h4c.si, _h4d.cfe, _h4d.cfs, _h4d.si, _h4e.cfe, 
_h4e.cfs, _h4e.si, _h4f.cfe, _h4f.cfs, _h4f.si, _h4g.cfe, _h4g.cfs, _h4g.si, 
_h4h.cfe, _h4h.cfs, _h4h.si, _h4i.cfe, _h4i.cfs, _h4i.si, _h4j.cfe, _h4j.cfs, 
_h4j.si, _h4k.cfe, _h4k.cfs, _h4k.si, _h4l.cfe, _h4l.cfs, _h4l.si, _h4m.cfe, 
_h4m.cfs, _h4m.si, _h4n.cfe, _h4n.cfs, _h4n.si, _hk9.cfe, _hk9.cfs, _hk9.si, 
_hka.cfe, _hka.cfs, _hka.si, _hkb.cfe, _hkb.cfs, _hkb.si, _hkc.cfe, _hkc.cfs, 
_hkc.si, _hkd.cfe, _hkd.cfs, _hkd.si, _hke.cfe, _hke.cfs, _hke.si, _hkf.cfe, 
_hkf.cfs, _hkf.si, _hkg.cfe, _hkg.cfs, _hkg.si, _hkh.cfe, _hkh.cfs, _hkh.si, 
_hki.cfe, _hki.cfs, _hki.si, _hkj.cfe, _hkj.cfs, _hkj.si, _hkk.cfe, _hkk.cfs, 
_hkk.si, _hkl.cfe, _hkl.cfs, _hkl.si, _hkm.cfe, _hkm.cfs, _hkm.si, _hkn.cfe, 
_hkn.cfs, _hkn.si, _hko.cfe, _hko.cfs, _hko.si, _hkp.cfe, _hkp.cfs, _hkp.si, 
_hkq.cfe, _hkq.cfs, _hkq.si, _hkr.cfe, _hkr.cfs, _hkr.si, _hks.cfe, _hks.cfs, 
_hks.si, _hkt.cfe, _hkt.cfs, _hkt.si, _hku.cfe, _hku.cfs, _hku.si, _hkv.cfe, 
_hkv.cfs, _hkv.si, _hkw.cfe, _hkw.cfs, _hkw.si, _hkx.cfe, _hkx.cfs, _hkx.si, 
_hky.cfe, _hky.cfs, _hky.si, _hkz.cfe, _hkz.cfs, _hkz.si, _hl0.cfe, _hl0.cfs, 
_hl0.si, _hl1.cfe, _hl1.cfs, _hl1.si, _hl2.cfe, _hl2.cfs, _hl2.si, _hl3.cfe, 
_hl3.cfs, _hl3.si, segments_2]}]> but 
was:<[{indexVersion=1461412769245,generation=2,filelist=[_go1.fdt, _go1.fdx, 
_go1.fnm, _go1.nvd, _go1.nvm, _go1.si, _go1_SimpleText_0.pst, _h3l.cfe, 
_h3l.cfs, _h3l.si, _h3m.cfe, _h3m.cfs, _h3m.si, _h3n.cfe, _h3n.cfs, _h3n.si, 
_h3o.cfe, _h3o.cfs, _h3o.si, _h3p.cfe, _h3p.cfs, _h3p.si, _h3q.cfe, _h3q.cfs, 
_h3q.si, _h3r.cfe, _h3r.cfs, _h3r.si, _h3s.cfe, _h3s.cfs, _h3s.si, _h3t.cfe, 
_h3t.cfs, _h3t.si, _h3u.cfe, _h3u.cfs, _h3u.si, _h3v.cfe, _h3v.cfs, _h3v.si, 
_h3w.cfe, _h3w.cfs, _h3w.si, _h3x.cfe, _h3x.cfs, _h3x.si, _h3y.cfe, _h3y.cfs, 
_h3y.si, _h3z.cfe, _h3z.cfs, _h3z.si, _h40.cfe, _h40.cfs, _h40.si, _h41.cfe, 
_h41.cfs, _h41.si, _h42.cfe, _h42.cfs, _h42.si, _h43.cfe, _h43.cfs, _h43.si, 
_h44.cfe, _h44.cfs, _h44.si, _h45.cfe, _h45.cfs, _h45.si, _h46.cfe, _h46.cfs, 
_h46.si, _h47.cfe, _h47.cfs, _h47.si, _h48.cfe, _h48.cfs, _h48.si, _h49.cfe, 
_h49.cfs, _h49.si, _h4a.cfe, _h4a.cfs, _h4a.si, _h4b.cfe, _h4b.cfs, _h4b.si, 
_h4c.cfe, _h4c.cfs, _h4c.si, _h4d.cfe, _h4d.cfs, _h4d.si, _h4e.cfe, _h4e.cfs, 
_h4e.si, _h4f.cfe, _h4f.cfs, _h4f.si, _h4g.cfe, _h4g.cfs, _h4g.si, _h4h.cfe, 
_h4h.cfs, _h4h.si, _h4i.cfe, _h4i.cfs, _h4i.si, _h4j.cfe, _h4j.cfs, _h4j.si, 
_h4k.cfe, _h4k.cfs, _h4k.si, _h4l.cfe, _h4l.cfs, _h4l.si, _h4m.cfe, _h4m.cfs, 
_h4m.si, _h4n.cfe, _h4n.cfs, _h4n.si, _hk9.cfe, _hk9.cfs, _hk9.si, _hka.cfe, 
_hka.cfs, _hka.si, _hkb.cfe, _hkb.cfs, _hkb.si, _hkc.cfe, _hkc.cfs, _hkc.si, 
_hkd.cfe, _hkd.cfs, _hkd.si, _hke.cfe, _hke.cfs, _hke.si, _hkf.cfe, _hkf.cfs, 
_hkf.si, _hkg.cfe, _hkg.cfs, _hkg.si, _hkh.cfe, _hkh.cfs, _hkh.si, _hki.cfe, 
_hki.cfs, _hki.si, _hkj.cfe, _hkj.cfs, _hkj.si, _hkk.cfe, _hkk.cfs, _hkk.si, 
_hkl.cfe, _hkl.cfs, _hkl.si, _hkm.cfe, _hkm.cfs, _hkm.si, _hkn.cfe, _hkn.cfs, 
_hkn.si, _hko.cfe, _hko.cfs, _hko.si, _hkp.cfe, _hkp.cfs, _hkp.si, _hkq.cfe, 
_hkq.cfs, _hkq.si, _hkr.cfe, _hkr.cfs, _hkr.si, _hks.cfe, _hks.cfs, _hks.si, 
_hkt.cfe, _hkt.cfs, _hkt.si, _hku.cfe, _hku.cfs, _hku.si, _hkv.cfe, _hkv.cfs, 
_hkv.si, _hkw.cfe, _hkw.cfs, _hkw.si, _hkx.cfe, _hkx.cfs, _hkx.si, _hky.cfe, 
_hky.cfs, _hky.si, _hkz.cfe, _hkz.cfs, _hkz.si, _hl0.cfe, _hl0.cfs, _hl0.si, 
_hl1.cfe, _hl1.cfs, _hl1.si, _hl2.cfe, _hl2.cfs, _hl2.si, _hl3.cfe, _hl3.cfs, 
_hl3.si, segments_2]}, 
{indexVersion=1461412769245,generation=3,filelist=[_h4q.fdt, _h4q.fdx, 
_h4q.fnm, _h4q.nvd, _h4q.nvm, _h4q.si, _h4q_SimpleText_0.pst, _hk9.cfe, 
_hk9.cfs, _hk9.si, _hka.cfe, _hka.cfs, _hka.si, _hkb.cfe, _hkb.cfs, _hkb.si, 
_hkc.cfe, _hkc.cfs, _hkc.si, _hkd.cfe, _hkd.cfs, _hkd.si, _hke.cfe, _hke.cfs, 
_hke.si, _hkf.cfe, _hkf.cfs, _hkf.si, _hkg.cfe, _hkg.cfs, _hkg.si, _hkh.cfe, 
_hkh.cfs, _hkh.si, _hki.cfe, _hki.cfs, _hki.si, _hkj.cfe, _hkj.cfs, _hkj.si, 
_hkk.cfe, _hkk.cfs, _hkk.si, _hkl.cfe, _hkl.cfs, _hkl.si, _hkm.cfe, _hkm.cfs, 
_hkm.si, _hkn.cfe, _hkn.cfs, _hkn.si, _hko.cfe, _hko.cfs, _hko.si, _hkp.cfe, 
_hkp.cfs, _hkp.si, _hkq.cfe, _hkq.cfs, _hkq.si, _hkr.cfe, _hkr.cfs, _hkr.si, 
_hks.cfe, _hks.cfs, _hks.si, _hkt.cfe, _hkt.cfs, _hkt.si, _hku.cfe, _hku.cfs, 
_hku.si, _hkv.cfe, _hkv.cfs, _hkv.si, _hkw.cfe, _hkw.cfs, _hkw.si, _hkx.cfe, 
_hkx.cfs, _hkx.si, _hky.cfe, _hky.cfs, _hky.si, _hkz.cfe, _hkz.cfs, _hkz.si, 
_hl0.cfe, _hl0.cfs, _hl0.si, _hl1.cfe, _hl1.cfs, _hl1.si, _hl2.cfe, _hl2.cfs, 
_hl2.si, _hl3.cfe, _hl3.cfs, _hl3.si, segments_3]}]>
        at 
__randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A:3EB54C5D8B15AC49]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:147)
        at 
org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
ObjectTracker found 48 object(s) that were not released!!! [InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 48 object(s) that were not 
released!!! [InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient]
        at __randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNull(Assert.java:551)
        at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:255)
        at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDistributedZkTest

Error Message:
48 threads leaked from SUITE scope at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest:     1) Thread[id=71460, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=68140, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=70719, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=73295, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=73469, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=67679, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=71462, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=73703, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=67681, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=73470, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=70718, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=71461, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=73701, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=69297, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=73296, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=69298, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=73702, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=67678, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=71463, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=69300, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=73012, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=73015, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=70717, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=71847, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=71180, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=68138, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=70716, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=71178, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=73013, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=71609, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=71608, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=71844, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=71848, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   34) Thread[id=71177, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=71179, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=73700, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=68139, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=69299, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=73468, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=67680, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=68137, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=73293, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=71845, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=71607, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=73014, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=73294, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=73467, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=71606, 
name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]         at 
java.lang.Thread.sleep(Native Method)         at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 48 threads leaked from 
SUITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest: 
   1) Thread[id=71460, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=68140, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=70719, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=73295, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=73469, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-CdcrReplicationDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConne

[...truncated too long message...]

enChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 10407702 WARN  
(zkCallback-2378-thread-29-processing-n:127.0.0.1:44395_solr) 
[n:127.0.0.1:44395_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 10407702 INFO  (jetty-launcher-2372-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@3c18bab{/solr,null,UNAVAILABLE}
   [junit4]   2> 10407709 INFO  
(TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[1B62576DFB5DA24A]) 
[    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:49088 49088
   [junit4]   2> 10407924 INFO  (Thread-15093) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:49088 49088
   [junit4]   2> 10407932 INFO  
(TEST-TestConfigSetsAPIExclusivity.testAPIExclusivity-seed#[1B62576DFB5DA24A]) 
[    ] o.a.s.SolrTestCaseJ4 ###Ending testAPIExclusivity
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestConfigSetsAPIExclusivity -Dtests.method=testAPIExclusivity 
-Dtests.seed=1B62576DFB5DA24A -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=sr-RS -Dtests.timezone=Indian/Reunion -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] ERROR   5102s J1 | TestConfigSetsAPIExclusivity.testAPIExclusivity 
<<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Unexpected exception: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:44395/solr: create the configset time out:300s 
expected:<0> but was:<66>
   [junit4]    >        at 
org.apache.solr.cloud.TestConfigSetsAPIExclusivity.testAPIExclusivity(TestConfigSetsAPIExclusivity.java:91)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)Throwable #2: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=28988, name=OverseerThreadFactory-8216-thread-4, 
state=RUNNABLE, group=Overseer collection creation process.]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit 
exceeded
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([1B62576DFB5DA24A]:0)
   [junit4]    >        at 
java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1420)
   [junit4]    >        at 
java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72)
   [junit4]    >        at 
java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:497)
   [junit4]    >        at 
java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
   [junit4]    >        at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]    >        at 
java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
   [junit4]    >        at 
java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
   [junit4]    >        at 
java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
   [junit4]    >        at 
java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
   [junit4]    >        at java.util.ArrayList.writeObject(ArrayList.java:762)
   [junit4]    >        at 
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
   [junit4]    >        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
   [junit4]    >        at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
   [junit4]    >        at 
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
   [junit4]   2> 10407936 INFO  
(SUITE-TestConfigSetsAPIExclusivity-seed#[1B62576DFB5DA24A]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/J1/temp/solr.cloud.TestConfigSetsAPIExclusivity_1B62576DFB5DA24A-001
   [junit4]   2> Apr 23, 2016 2:14:04 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60), 
sim=ClassicSimilarity, locale=sr-RS, timezone=Indian/Reunion
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_74 (64-bit)/cpus=4,threads=1,free=86820624,total=530055168
   [junit4]   2> NOTE: All tests run in this JVM: [SortByFunctionTest, 
TestRandomCollapseQParserPlugin, DistributedIntervalFacetingTest, 
TestCodecSupport, SoftAutoCommitTest, DistributedFacetPivotSmallTest, 
ConvertedLegacyTest, SpellPossibilityIteratorTest, 
HdfsChaosMonkeyNothingIsSafeTest, TestManagedSchemaAPI, 
SpellCheckComponentTest, SolrXmlInZkTest, OverseerRolesTest, 
ChaosMonkeyNothingIsSafeTest, MoreLikeThisHandlerTest, BadCopyFieldTest, 
TestSolrCLIRunExample, BlockJoinFacetSimpleTest, IndexBasedSpellCheckerTest, 
TestShardHandlerFactory, SaslZkACLProviderTest, AnalysisErrorHandlingTest, 
DistributedTermsComponentTest, TestPartialUpdateDeduplication, 
ConjunctionSolrSpellCheckerTest, TestRequestStatusCollectionAPI, 
CoreMergeIndexesAdminHandlerTest, TimeZoneUtilsTest, TestDefaultStatsCache, 
TestTrieFacet, DeleteReplicaTest, HdfsNNFailoverTest, DistanceFunctionTest, 
CollectionReloadTest, TestRestManager, SchemaVersionSpecificBehaviorTest, 
SpatialFilterTest, ZkStateWriterTest, TestSolrCoreProperties, 
WordBreakSolrSpellCheckerTest, TestSolrFieldCacheMBean, TestSolrXml, 
TestScoreJoinQPScore, TestFieldTypeCollectionResource, SolrInfoMBeanTest, 
OpenCloseCoreStressTest, TestTrie, SuggesterFSTTest, 
TestMiniSolrCloudClusterSSL, TestCopyFieldCollectionResource, 
OpenExchangeRatesOrgProviderTest, DocumentBuilderTest, 
TestSortingResponseWriter, ShardRoutingCustomTest, SolrPluginUtilsTest, 
ClusterStateUpdateTest, DistributedQueryComponentOptimizationTest, 
TestBinaryResponseWriter, ConfigSetsAPITest, TestCoreContainer, 
SimplePostToolTest, MultiTermTest, CollectionsAPIDistributedZkTest, 
TestBulkSchemaConcurrent, WrapperMergePolicyFactoryTest, CustomCollectionTest, 
TestConfigSetsAPI, VersionInfoTest, FileUtilsTest, 
IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestMiniSolrCloudCluster, 
AsyncMigrateRouteKeyTest, TestSolrDeletionPolicy2, TestSurroundQueryParser, 
TestQuerySenderNoQuery, ZkSolrClientTest, TestDynamicLoading, 
LoggingHandlerTest, LeaderInitiatedRecoveryOnShardRestartTest, HighlighterTest, 
NotRequiredUniqueKeyTest, TestCloudSchemaless, TestDFRSimilarityFactory, 
ClassificationUpdateProcessorFactoryTest, TestFileDictionaryLookup, 
RecoveryAfterSoftCommitTest, TestConfigSetImmutable, 
DirectSolrSpellCheckerTest, UniqFieldsUpdateProcessorFactoryTest, 
ShardRoutingTest, BigEndianAscendingWordSerializerTest, SparseHLLTest, 
DistributedFacetPivotLargeTest, HdfsRestartWhileUpdatingTest, 
DistributedQueueTest, RestartWhileUpdatingTest, 
TestLMJelinekMercerSimilarityFactory, TestConfigSetsAPIExclusivity]
   [junit4] Completed [590/590 (12!)] on J1 in 5102.60s, 1 test, 1 error <<< 
FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build/solr-core/test/temp/junit4-J1-20160423_112030_672.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/heapdumps/java_pid26042.hprof
 ...
   [junit4] Heap dump file created [706821575 bytes in 11.976 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 19 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:747:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:684:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/solr/common-build.xml:529:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1457:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/lucene/common-build.xml:1014:
 There were test failures: 590 suites (10 ignored), 2384 tests, 9 suite-level 
errors, 8 errors, 2 failures, 31 ignored (16 assumptions) [seed: 
1B62576DFB5DA24A]

Total time: 408 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org