Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/856/

3 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:46164/awholynewcollection_0: Expected 
mime type application/octet-stream but got text/html. <html> <head> <meta 
http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 
500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing 
/awholynewcollection_0/select. Reason: <pre>    {msg=Error trying to proxy 
request for url: 
http://127.0.0.1:46308/awholynewcollection_0/select,trace=org.apache.solr.common.SolrException:
 Error trying to proxy request for url: 
http://127.0.0.1:46308/awholynewcollection_0/select  at 
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:596)  at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
  at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)  
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) 
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
at org.eclipse.jetty.server.Server.handle(Server.java:499)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)  at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
 at java.lang.Thread.run(Thread.java:745) Caused by: 
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for 
connection from pool  at 
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)
  at 
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
  at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
  at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
  at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:563)  
... 24 more ,code=500}</pre></p> <hr /><i><small>Powered by 
Jetty://</small></i> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:46164/awholynewcollection_0: Expected mime type 
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /awholynewcollection_0/select. Reason:
<pre>    {msg=Error trying to proxy request for url: 
http://127.0.0.1:46308/awholynewcollection_0/select,trace=org.apache.solr.common.SolrException:
 Error trying to proxy request for url: 
http://127.0.0.1:46308/awholynewcollection_0/select
        at 
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:596)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting 
for connection from pool
        at 
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)
        at 
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at 
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:563)
        ... 24 more
,code=500}</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

        at 
__randomizedtesting.SeedInfo.seed([BA78D4A59A09DEBB:322CEB7F34F5B343]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:543)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
        at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForNon403or404or503(AbstractFullDistribZkTestBase.java:1753)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:653)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:155)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest:     1) 
Thread[id=6897, name=zkCallback-880-thread-2, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=6898, 
name=zkCallback-880-thread-3, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=6439, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-SendThread(127.0.0.1:56190),
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         
at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    4) 
Thread[id=6441, name=zkCallback-880-thread-1, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    5) Thread[id=6440, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-EventThread,
 state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)     
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest: 
   1) Thread[id=6897, name=zkCallback-880-thread-2, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=6898, name=zkCallback-880-thread-3, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=6439, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-SendThread(127.0.0.1:56190),
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   4) Thread[id=6441, name=zkCallback-880-thread-1, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=6440, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-EventThread,
 state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
        at __randomizedtesting.SeedInfo.seed([BA78D4A59A09DEBB]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) 
Thread[id=6897, name=zkCallback-880-thread-2, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    2) Thread[id=6898, 
name=zkCallback-880-thread-3, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)    3) Thread[id=6439, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-SendThread(127.0.0.1:56190),
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         
at java.lang.Thread.sleep(Native Method)         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)    
     at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    
4) Thread[id=6441, name=zkCallback-880-thread-1, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at 
sun.misc.Unsafe.park(Native Method)         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
      at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
        at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=6897, name=zkCallback-880-thread-2, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=6898, name=zkCallback-880-thread-3, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=6439, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-SendThread(127.0.0.1:56190),
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   4) Thread[id=6441, name=zkCallback-880-thread-1, state=TIMED_WAITING, 
group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        at __randomizedtesting.SeedInfo.seed([BA78D4A59A09DEBB]:0)




Build Log:
[...truncated 10342 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/init-core-data-001
   [junit4]   2> 909088 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 909088 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 909199 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 909206 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 909209 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 909227 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_41906_hdfs____q236z0/webapp
   [junit4]   2> 909341 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log NO JSP Support for /, did not find 
org.apache.jasper.servlet.JspServlet
   [junit4]   2> 909583 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41906
   [junit4]   2> 909851 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 909852 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 909876 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_49888_datanode____uwenlm/webapp
   [junit4]   2> 909979 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log NO JSP Support for /, did not find 
org.apache.jasper.servlet.JspServlet
   [junit4]   2> 910148 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49888
   [junit4]   2> 910209 WARN  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 910210 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log jetty-6.1.26
   [junit4]   2> 910221 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_39245_datanode____.m1idn0/webapp
   [junit4]   2> 910348 INFO  (IPC Server handler 4 on 51199) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-2bf31472-b579-44db-84d7-1d9938d5379e node DatanodeRegistration(127.0.0.1, 
datanodeUuid=d2c02272-a86e-4815-8ab7-33832405d147, infoPort=49888, 
ipcPort=45341, storageInfo=lv=-56;cid=testClusterID;nsid=922711261;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 910348 INFO  (IPC Server handler 4 on 51199) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-bff17d16-3f4b-4d74-8e7d-926d413e113b node DatanodeRegistration(127.0.0.1, 
datanodeUuid=d2c02272-a86e-4815-8ab7-33832405d147, infoPort=49888, 
ipcPort=45341, storageInfo=lv=-56;cid=testClusterID;nsid=922711261;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 910368 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log NO JSP Support for /, did not find 
org.apache.jasper.servlet.JspServlet
   [junit4]   2> 911461 INFO  
(SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[BA78D4A59A09DEBB]-worker) [    
] o.m.log Started 
HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39245
   [junit4]   2> 911592 INFO  (IPC Server handler 6 on 51199) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-6fbb9356-30a1-44ac-9d3f-7f13177d6e14 node DatanodeRegistration(127.0.0.1, 
datanodeUuid=ba957c5f-9d77-4959-bddd-ac3ae1a199cf, infoPort=39245, 
ipcPort=53673, storageInfo=lv=-56;cid=testClusterID;nsid=922711261;c=0), 
blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 911592 INFO  (IPC Server handler 6 on 51199) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-7d9429ba-13be-418e-8144-71fa0ece884f node DatanodeRegistration(127.0.0.1, 
datanodeUuid=ba957c5f-9d77-4959-bddd-ac3ae1a199cf, infoPort=39245, 
ipcPort=53673, storageInfo=lv=-56;cid=testClusterID;nsid=922711261;c=0), 
blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 911654 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 911655 INFO  (Thread-2597) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 911655 INFO  (Thread-2597) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 911755 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:56190
   [junit4]   2> 911755 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 911755 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 911758 INFO  (zkCallback-849-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5a0ac646 
name:ZooKeeperConnection Watcher:127.0.0.1:56190 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 911758 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 911759 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 911759 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 911762 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 911763 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 911764 INFO  (zkCallback-850-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@663c1278 
name:ZooKeeperConnection Watcher:127.0.0.1:56190/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 911764 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 911764 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 911764 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 911767 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 911768 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 911769 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 911771 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 911771 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 911773 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 911774 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 911776 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 911776 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 911778 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 911778 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 911779 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 911780 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 911781 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 911781 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 911783 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 911783 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 911784 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 911785 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 911786 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 911786 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 911788 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 911788 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 911790 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 911790 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 912088 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores/collection1
   [junit4]   2> 912089 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 912091 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@5180c56f{/,null,AVAILABLE}
   [junit4]   2> 912092 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.e.j.s.ServerConnector Started 
ServerConnector@6d9904b7{HTTP/1.1}{127.0.0.1:47950}
   [junit4]   2> 912092 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.e.j.s.Server Started @915001ms
   [junit4]   2> 912092 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=hdfs://localhost:51199/hdfs__localhost_51199__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-trunk_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001_tempDir-002_control_data,
 hostContext=/, hostPort=47950, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores}
   [junit4]   2> 912092 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 912092 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/'
   [junit4]   2> 912092 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 912093 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 912108 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 912108 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 912109 INFO  (zkCallback-851-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6daad99a 
name:ZooKeeperConnection Watcher:127.0.0.1:56190/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 912109 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 912110 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 912111 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 912111 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/solr.xml
   [junit4]   2> 912117 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores
   [junit4]   2> 912117 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.CoreContainer New CoreContainer 1021446176
   [junit4]   2> 912117 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/]
   [junit4]   2> 912117 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/lib
   [junit4]   2> 912117 WARN  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/lib).
   [junit4]   2> 912124 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme 
: ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 
10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 
5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 912125 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 912126 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 912126 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 912126 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56190/solr
   [junit4]   2> 912126 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 912126 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 912127 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 912134 INFO  (zkCallback-853-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@5d2d500 name:ZooKeeperConnection 
Watcher:127.0.0.1:56190 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 912135 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 912135 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 912137 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 912138 INFO  
(zkCallback-854-thread-1-processing-n:127.0.0.1:47950_) [n:127.0.0.1:47950_    
] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@8c217cc name:ZooKeeperConnection 
Watcher:127.0.0.1:56190/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 912138 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 912139 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 912140 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 912142 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 912144 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 912145 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 912147 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 912148 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 912149 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 912150 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 912151 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:47950_
   [junit4]   2> 912152 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:47950_
   [junit4]   2> 912153 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 912154 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 912155 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 912155 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.LeaderElector Joined leadership election with 
path: /overseer_elect/election/94885719969431556-127.0.0.1:47950_-n_0000000000
   [junit4]   2> 912156 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:47950_
   [junit4]   2> 912156 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 912157 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.Overseer Overseer 
(id=94885719969431556-127.0.0.1:47950_-n_0000000000) starting
   [junit4]   2> 912158 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 912163 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=10000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 912164 INFO  
(OverseerCollectionConfigSetProcessor-94885719969431556-127.0.0.1:47950_-n_0000000000)
 [n:127.0.0.1:47950_    ] o.a.s.c.OverseerTaskProcessor Process current queue 
of overseer operations
   [junit4]   2> 912164 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 912164 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 912183 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 912183 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 912183 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores
   [junit4]   2> 912184 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores/collection1,
 collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 912184 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores/collection1/
   [junit4]   2> 912184 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 912185 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 912185 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) 
[n:127.0.0.1:47950_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() 
done
   [junit4]   2> 912186 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 912186 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 912187 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:47950";,
   [junit4]   2>   "node_name":"127.0.0.1:47950_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 912188 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.ZkController 
look for our core node name
   [junit4]   2> 912188 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:47950";,
   [junit4]   2>   "node_name":"127.0.0.1:47950_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 912188 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
control_collection
   [junit4]   2> 912189 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard 
shard=shard1
   [junit4]   2> 912190 INFO  
(zkCallback-854-thread-1-processing-n:127.0.0.1:47950_) [n:127.0.0.1:47950_    
] o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 913188 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.ZkController 
waiting to find shard id in clusterstate for collection1
   [junit4]   2> 913188 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Check for collection zkNode:control_collection
   [junit4]   2> 913189 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 913189 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 913189 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 913189 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores/collection1/'
   [junit4]   2> 913189 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 913190 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 913212 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.Config loaded 
config solrconfig.xml with version 0 
   [junit4]   2> 913218 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 913222 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 913244 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.Config Loaded 
SolrConfig: solrconfig.xml
   [junit4]   2> 913245 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 913276 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 913417 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 913424 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
default search field in schema is text
   [junit4]   2> 913425 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.s.IndexSchema 
unique key field: id
   [junit4]   2> 913438 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 913440 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 913442 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 913453 WARN  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 913453 WARN  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 913453 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 913454 WARN  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 913454 WARN  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 913455 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 913455 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection   x:collection1] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1' using configuration from collection 
control_collection
   [junit4]   2> 913455 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 913455 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory 
solr.hdfs.home=hdfs://localhost:51199/solr_hdfs_home
   [junit4]   2> 913456 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 913456 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_BA78D4A59A09DEBB-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 913456 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@367fe632
   [junit4]   2> 913456 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:51199/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 913472 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:51199/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 913473 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore New index directory detected: old=null 
new=hdfs://localhost:51199/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 913499 WARN  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore [collection1] Solr index directory 
'hdfs:/localhost:51199/solr_hdfs_home/control_collection/core_node1/data/index' 
doesn't exist. Creating new index...
   [junit4]   2> 913499 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:51199/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 913520 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory return new directory for 
hdfs://localhost:51199/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 913520 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=48, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.3271484375, noCFSRatio=0.0]
   [junit4]   2> 913575 INFO  (IPC Server handler 5 on 51199) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:52975 is 
added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, 
primaryNodeIndex=-1, 
replicas=[ReplicaUnderConstruction[[DISK]DS-6fbb9356-30a1-44ac-9d3f-7f13177d6e14:NORMAL:127.0.0.1:36932|RBW],
 
ReplicaUnderConstruction[[DISK]DS-2bf31472-b579-44db-84d7-1d9938d5379e:NORMAL:127.0.0.1:52975|RBW]]}
 size 0
   [junit4]   2> 913578 INFO  (IPC Server handler 3 on 51199) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36932 is 
added to blk_1073741825_1001 size 71
   [junit4]   2> 913582 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        commit{dir=NRTCachingDirectory(HdfsDirectory@b81bf0d9 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17ec927d; 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 913582 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 913584 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
Looking for old index directories to cleanup for core collection1 in 
hdfs://localhost:51199/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 913587 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"nodistrib"
   [junit4]   2> 913587 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"dedupe"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"stored_sig"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-explicit"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 913588 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating 
implicit default
   [junit4]   2> 913588 WARN  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 913590 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 913591 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 913592 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 913593 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 913596 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/dump,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 913596 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 913597 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 913597 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Initializing HdfsUpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 tlogDfsReplication=2
   [junit4]   2> 913613 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 913613 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 913614 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=11, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=30.908203125, 
floorSegmentMB=1.2041015625, forceMergeDeletesPctAllowed=5.108906690701219, 
segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 913628 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        commit{dir=NRTCachingDirectory(HdfsDirectory@b81bf0d9 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17ec927d; 
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 913628 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 913628 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.s.SolrIndexSearcher Opening Searcher@275bf12[collection1] main
   [junit4]   2> 913629 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 913629 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 913630 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the 
RestManager with znodeBase: /configs/conf1
   [junit4]   2> 913630 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 913630 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 913630 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 913630 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 913630 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 913631 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 913631 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 913632 INFO  
(searcherExecutor-2563-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Registered new searcher Searcher@275bf12[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 913632 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 913632 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo Refreshing highest value of _version_ for 256 version 
buckets from index
   [junit4]   2> 913632 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket 
highest value from index
   [junit4]   2> 913632 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1518171521582891008
   [junit4]   2> 913632 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with highest version 
1518171521582891008
   [junit4]   2> 913633 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 913633 INFO  
(coreLoadExecutor-2562-thread-1-processing-n:127.0.0.1:47950_) 
[n:127.0.0.1:47950_ c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 913634 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
Register replica - core:collection1 address:http://127.0.0.1:47950 
collection:control_collection shard:shard1
   [junit4]   2> 913635 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 913639 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector 
Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/94885719969431556-core_node1-n_0000000000
   [junit4]   2> 913639 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 913641 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 913641 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 913641 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:47950/collection1/
   [junit4]   2> 913641 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 913641 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 913641 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:47950/collection1/ has no replicas
   [junit4]   2> 913642 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 913644 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 913646 INFO  
(coreZkRegister-2556-thread-1-processing-n:127.0.0.1:47950_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:47950_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:47950/collection1/ shard1
   [junit4]   2> 913647 INFO  
(OverseerStateUpdate-94885719969431556-127.0.0.1:47950_-n_0000000000) 
[n:127.0.0.1:47950_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:47950";,
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 913687 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 913688 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 913689 INFO  (zkCallback-856-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4d49c3ae 
name:ZooKeeperConnection Watcher:127.0.0.1:56190/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 913689 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 913689 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 913689 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 913691 INFO  
(TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection los

[...truncated too long message...]

ile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_CR 
-Dtests.timezone=Europe/Zurich -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | HdfsCollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest: 
   [junit4]    >    1) Thread[id=6897, name=zkCallback-880-thread-2, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=6898, name=zkCallback-880-thread-3, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=6439, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-SendThread(127.0.0.1:56190),
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    4) Thread[id=6441, name=zkCallback-880-thread-1, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    5) Thread[id=6440, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-EventThread,
 state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([BA78D4A59A09DEBB]:0)Throwable #2: 
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=6897, name=zkCallback-880-thread-2, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    2) Thread[id=6898, name=zkCallback-880-thread-3, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >    3) Thread[id=6439, 
name=TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[BA78D4A59A09DEBB]-SendThread(127.0.0.1:56190),
 state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    4) Thread[id=6441, name=zkCallback-880-thread-1, 
state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([BA78D4A59A09DEBB]:0)
   [junit4] Completed [184/558] on J2 in 104.52s, 1 test, 3 errors <<< FAILURES!

[...truncated 1213 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 558 suites (8 ignored), 2215 tests, 2 suite-level 
errors, 1 error, 1078 ignored (9 assumptions) [seed: BA78D4A59A09DEBB]

Total time: 219 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to