Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java8/14/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=9368, 
name=parallelCoreAdminExecutor-4389-thread-15, state=RUNNABLE, 
group=TGRP-MultiThreadedOCPTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=9368, 
name=parallelCoreAdminExecutor-4389-thread-15, state=RUNNABLE, 
group=TGRP-MultiThreadedOCPTest]
        at 
__randomizedtesting.SeedInfo.seed([43F7A82DEDFEC004:CBA397F74302ADFC]:0)
Caused by: java.lang.AssertionError: Too many closes on SolrCore
        at __randomizedtesting.SeedInfo.seed([43F7A82DEDFEC004]:0)
        at org.apache.solr.core.SolrCore.close(SolrCore.java:1150)
        at org.apache.solr.common.util.IOUtils.closeQuietly(IOUtils.java:31)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:652)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:611)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:628)
        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
        at 
org.apache.solr.handler.admin.CoreAdminHandler$ParallelCoreAdminHandlerThread.run(CoreAdminHandler.java:1249)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr

Error Message:
No live SolrServers available to handle this 
request:[http://127.0.0.1:49735/vead/p]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:49735/vead/p]
        at 
__randomizedtesting.SeedInfo.seed([43F7A82DEDFEC004:E8EE7EEAA79F0033]:0)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
        at 
org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:165)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:49735/vead/p: Expected mime type 
application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /vead/p/admin/collections. Reason:
<pre>    org.apache.solr.common.SolrException: Error during request 
authentication, </pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
        ... 47 more




Build Log:
[...truncated 10456 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberos
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/init-core-data-001
   [junit4]   2> 1520796 T9898 
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system 
property: /vead/p
   [junit4]   2> 1523681 T9898 
oadsc.DefaultDirectoryService.showSecurityWarnings WARN You didn't change the 
admin password of directory service instance 'DefaultKrbServer'.  Please update 
the admin password as soon as possible to prevent a possible security breach.
   [junit4]   2> 1523972 T9898 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1523973 T9906 oasc.ZkTestServer$2$1.setClientPort client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1523973 T9906 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 1524073 T9898 oasc.ZkTestServer.run start zk server on 
port:50190
   [junit4]   2> 1524082 T9913 oascc.ConnectionManager.process WARN zkClient 
received AuthFailed
   [junit4]   2> 1524086 T9916 oascc.ConnectionManager.process WARN zkClient 
received AuthFailed
   [junit4]   2> 1524091 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1524093 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1524095 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1524097 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1524098 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1524100 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1524101 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1524102 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1524104 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1524105 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1524107 T9898 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1524109 T9919 oascc.ConnectionManager.process WARN zkClient 
received AuthFailed
   [junit4]   2> 1524403 T9898 oas.SolrTestCaseJ4.writeCoreProperties Writing 
core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1
   [junit4]   2> 1524405 T9898 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1524406 T9898 oejsh.ContextHandler.doStart Started 
o.e.j.s.ServletContextHandler@795d0f8a{/vead/p,null,AVAILABLE}
   [junit4]   2> 1524407 T9898 oejs.AbstractConnector.doStart Started 
ServerConnector@350f28a6{HTTP/1.1}{127.0.0.1:49735}
   [junit4]   2> 1524407 T9898 oejs.Server.doStart Started @1526182ms
   [junit4]   2> 1524407 T9898 oascse.JettySolrRunner$1.lifeCycleStarted Jetty 
properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-002/control/data, hostContext=/vead/p, 
hostPort=49735, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores}
   [junit4]   2> 1524408 T9898 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1524408 T9898 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/'
   [junit4]   2> 1524421 T9898 oasc.SolrXmlConfig.fromFile Loading container 
configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/solr.xml
   [junit4]   2> 1524453 T9898 oasc.CorePropertiesLocator.<init> Config-defined 
core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores
   [junit4]   2> 1524453 T9898 oasc.CoreContainer.<init> New CoreContainer 
1440517004
   [junit4]   2> 1524453 T9898 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/]
   [junit4]   2> 1524454 T9898 oasc.CoreContainer.load loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/lib
   [junit4]   2> 1524454 T9898 oasc.SolrResourceLoader.addToClassLoader WARN 
Can't find (or read) directory to add to classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/lib).
   [junit4]   2> 1524460 T9898 oashc.HttpShardHandlerFactory.init created with 
socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 1524461 T9898 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1524462 T9898 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1524462 T9898 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1524463 T9898 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1524463 T9898 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:50190/solr
   [junit4]   2> 1524463 T9898 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1524465 T9933 oascc.ConnectionManager.process WARN zkClient 
received AuthFailed
   [junit4]   2> 1524468 T9937 n:127.0.0.1:49735_vead%2Fp 
oascc.ConnectionManager.process WARN zkClient received AuthFailed
   [junit4]   2> 1524478 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.ZkController.createEphemeralLiveNode Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:49735_vead%2Fp
   [junit4]   2> 1524482 T9898 n:127.0.0.1:49735_vead%2Fp oasc.Overseer.close 
Overseer (id=null) closing
   [junit4]   2> 1524483 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 
127.0.0.1:49735_vead%2Fp
   [junit4]   2> 1524484 T9898 n:127.0.0.1:49735_vead%2Fp oasc.Overseer.start 
Overseer (id=93889084884975620-127.0.0.1:49735_vead%2Fp-n_0000000000) starting
   [junit4]   2> 1524490 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.OverseerAutoReplicaFailoverThread.<init> Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1524491 T9940 n:127.0.0.1:49735_vead%2Fp 
oasc.OverseerCollectionProcessor.run Process current queue of collection 
creations
   [junit4]   2> 1524491 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1524501 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.CoreContainer.initializeAuthenticationPlugin Authentication plugin class 
obtained from ZK: org.apache.solr.security.KerberosPlugin
   [junit4]   2> 1524511 T9898 n:127.0.0.1:49735_vead%2Fp 
oass.KerberosPlugin.init Params: {token.valid=30, 
kerberos.principal=HTTP/127.0.0.1, cookie.domain=127.0.0.1, 
kerberos.keytab=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/keytabs, cookie.path=/, 
type=kerberos, kerberos.name.rules=DEFAULT}
   [junit4]   2> 1524521 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.CoreContainer.initializeAuthenticationPlugin Reconfiguring the shard 
handler factory and update shard handler.
   [junit4]   2> 1524521 T9898 n:127.0.0.1:49735_vead%2Fp 
oashc.HttpShardHandlerFactory.reconfigureHttpClient Reconfiguring the default 
client with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@7d9e156c
   [junit4]   2> 1524521 T9898 n:127.0.0.1:49735_vead%2Fp 
oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1524524 T9898 n:127.0.0.1:49735_vead%2Fp 
oasu.UpdateShardHandler.reconfigureHttpClient Reconfiguring the default client 
with: org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer@7d9e156c
   [junit4]   2> 1524524 T9898 n:127.0.0.1:49735_vead%2Fp 
oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1524526 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.CorePropertiesLocator.discover Looking for core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores
   [junit4]   2> 1524527 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, 
config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1, 
collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/, coreNodeName=, 
dataDir=data/, shard=}
   [junit4]   2> 1524527 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.CorePropertiesLocator.discoverUnder Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/
   [junit4]   2> 1524527 T9898 n:127.0.0.1:49735_vead%2Fp 
oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1524528 T9943 n:127.0.0.1:49735_vead%2Fp c:control_collection 
x:collection1 oasc.ZkController.publish publishing core=collection1 state=down 
collection=control_collection
   [junit4]   2> 1524528 T9943 n:127.0.0.1:49735_vead%2Fp c:control_collection 
x:collection1 oasc.ZkController.publish numShards not found on descriptor - 
reading it from system property
   [junit4]   2> 1524529 T9937 n:127.0.0.1:49735_vead%2Fp 
oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path 
/overseer/queue state SyncConnected
   [junit4]   2> 1524529 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1524529 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "node_name":"127.0.0.1:49735_vead%2Fp",
   [junit4]   2>          "numShards":"1",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":null,
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"} current state version: 0
   [junit4]   2> 1524530 T9939 n:127.0.0.1:49735_vead%2Fp 
oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "node_name":"127.0.0.1:49735_vead%2Fp",
   [junit4]   2>          "numShards":"1",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":null,
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 1524530 T9939 n:127.0.0.1:49735_vead%2Fp 
oasco.ClusterStateMutator.createCollection building a new cName: 
control_collection
   [junit4]   2> 1524530 T9939 n:127.0.0.1:49735_vead%2Fp 
oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1525529 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for 
collection1
   [junit4]   2> 1525530 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.ZkController.createCollectionZkNode Check for collection 
zkNode:control_collection
   [junit4]   2> 1525530 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1525532 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/'
   [junit4]   2> 1525543 T9943 n:127.0.0.1:49735_vead%2Fp oasc.Config.<init> 
loaded config solrconfig.xml with version 0 
   [junit4]   2> 1525547 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1525550 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1525558 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1525559 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1525564 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1525654 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider.init Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1525659 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1525660 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1525668 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.FileExchangeRateProvider.reload Reloading exchange rates from file 
currency.xml
   [junit4]   2> 1525670 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.FileExchangeRateProvider.reload Reloading exchange rates from file 
currency.xml
   [junit4]   2> 1525672 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 1525673 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 1525673 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 1525673 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 1525674 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 1525674 T9943 n:127.0.0.1:49735_vead%2Fp 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 1525674 T9943 n:127.0.0.1:49735_vead%2Fp 
oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration 
from collection control_collection
   [junit4]   2> 1525674 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1525675 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1525675 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to 
JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ee5eb3a
   [junit4]   2> 1525675 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.CachingDirectoryFactory.get return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data
   [junit4]   2> 1525675 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.getNewIndexDir New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1525676 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.initIndex WARN [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index' doesn't exist. 
Creating new index...
   [junit4]   2> 1525676 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.CachingDirectoryFactory.get return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index
   [junit4]   2> 1525676 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=1.0]
   [junit4]   2> 1525677 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>                
commit{dir=MockDirectoryWrapper(RAMDirectory@6bf4d2f4 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7b73dfc4),segFN=segments_1,generation=1}
   [junit4]   2> 1525677 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1525679 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain 
"nodistrib"
   [junit4]   2> 1525680 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain 
"dedupe"
   [junit4]   2> 1525680 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1525680 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain 
"stored_sig"
   [junit4]   2> 1525680 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1525681 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain 
"distrib-dup-test-chain-explicit"
   [junit4]   2> 1525681 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 1525681 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 1525681 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined 
as default, creating implicit default
   [junit4]   2> 1525683 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1525684 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1525684 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1525685 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1525688 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.RequestHandlers.initHandlersFromConfig Registered paths: 
/admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1525688 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.initStatsCache Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1525689 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.UpdateHandler.<init> Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1525689 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1525690 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1525690 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1525690 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=29, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=31.8046875, 
floorSegmentMB=2.134765625, forceMergeDeletesPctAllowed=28.50146737269296, 
segmentsPerTier=22.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1525691 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>                
commit{dir=MockDirectoryWrapper(RAMDirectory@6bf4d2f4 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7b73dfc4),segFN=segments_1,generation=1}
   [junit4]   2> 1525691 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1525691 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oass.SolrIndexSearcher.<init> Opening Searcher@1f39f757[collection1] main
   [junit4]   2> 1525692 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for 
the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1525693 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured 
ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1525693 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1525693 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.ManagedResourceStorage.load Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1525694 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found 
for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1525694 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1525694 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1525694 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1525694 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.ZkController.getConfDirListeners watch zkdir /configs/conf1
   [junit4]   2> 1525695 T9944 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.SolrCore.registerSearcher [collection1] Registered new searcher 
Searcher@1f39f757[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1525695 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.UpdateLog.onFirstSearcher On first searcher opened, looking up max value 
of version field
   [junit4]   2> 1525695 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.VersionInfo.getMaxVersionFromIndex Refreshing highest value of _version_ 
for 65536 version buckets from index
   [junit4]   2> 1525695 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.VersionInfo.getMaxVersionFromIndex No terms found for _version_, cannot 
seed version bucket highest value from index
   [junit4]   2> 1525695 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.UpdateLog.seedBucketsWithHighestVersion Could not find max version in 
index or recent updates, using new clock 1502225359954771968
   [junit4]   2> 1525697 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasu.UpdateLog.seedBucketsWithHighestVersion Took 2 ms to seed version buckets 
with highest version 1502225359954771968
   [junit4]   2> 1525697 T9943 n:127.0.0.1:49735_vead%2Fp x:collection1 
oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1525698 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ZkController.register Register replica - 
core:collection1 address:http://127.0.0.1:49735/vead/p 
collection:control_collection shard:shard1
   [junit4]   2> 1525698 T9898 n:127.0.0.1:49735_vead%2Fp 
oascsi.Krb5HttpClientConfigurer.configure Setting up SPNego auth with config: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1525699 T9898 n:127.0.0.1:49735_vead%2Fp 
oass.SolrDispatchFilter.init 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1
   [junit4]   2> 1525699 T9898 n:127.0.0.1:49735_vead%2Fp 
oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1525700 T9898 oascsi.Krb5HttpClientConfigurer.configure 
Setting up SPNego auth with config: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1525700 T9898 oascsi.Krb5HttpClientConfigurer.configure 
Setting up SPNego auth with config: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1525701 T9950 oascc.ConnectionManager.process WARN zkClient 
received AuthFailed
   [junit4]   2> 1525702 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess Running 
the leader process for shard shard1
   [junit4]   2> 1525703 T9937 n:127.0.0.1:49735_vead%2Fp 
oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path 
/overseer/queue state SyncConnected
   [junit4]   2> 1525704 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp 
Enough replicas found to continue.
   [junit4]   2> 1525704 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2>          "operation":"leader",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"control_collection"} current state 
version: 1
   [junit4]   2> 1525704 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I may 
be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C996 name=collection1 
org.apache.solr.core.SolrCore@7cece221 
url=http://127.0.0.1:49735/vead/p/collection1 node=127.0.0.1:49735_vead%2Fp 
C996_STATE=coll:control_collection core:collection1 props:{core=collection1, 
base_url=http://127.0.0.1:49735/vead/p, node_name=127.0.0.1:49735_vead%2Fp, 
state=down}
   [junit4]   2> 1525705 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 C996 oasc.SyncStrategy.sync Sync replicas to 
http://127.0.0.1:49735/vead/p/collection1/
   [junit4]   2> 1525705 T9898 oas.SolrTestCaseJ4.setUp ###Starting 
testKerberizedSolr
   [junit4]   2> 1525705 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 C996 oasc.SyncStrategy.syncReplicas Sync Success - now 
sync replicas to me
   [junit4]   2> 1525705 T9898 oascsi.Krb5HttpClientConfigurer.configure 
Setting up SPNego auth with config: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/jaas-client.conf
   [junit4]   2> 1525706 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 C996 oasc.SyncStrategy.syncToMe 
http://127.0.0.1:49735/vead/p/collection1/ has no replicas
   [junit4]   2> 1525706 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ShardLeaderElectionContext.runLeaderProcess I am 
the new leader: http://127.0.0.1:49735/vead/p/collection1/ shard1
   [junit4]   2> 1525707 T9953 oascc.ConnectionManager.process WARN zkClient 
received AuthFailed
   [junit4]   2> 1525709 T9937 n:127.0.0.1:49735_vead%2Fp 
oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path 
/overseer/queue state SyncConnected
   [junit4]   2> 1525710 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2>          "operation":"leader",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "state":"active"} current state version: 1
   [junit4]   2> 1525719 T9898 oahcp.ResponseProcessCookies.processCookies WARN 
Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; 
Expires=to, 01-jan.-1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: 
to, 01-jan.-1970 00:00:00 GMT
   [junit4]   1> Debug is  true storeKey true useTicketCache false useKeyTab 
true doNotPrompt false ticketCache is null isInitiator true KeyTab is 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/tempDir-001/minikdc/keytabs refreshKrb5Config is true 
principal is HTTP/127.0.0.1 tryFirstPass is false useFirstPass is false 
storePass is false clearPass is false
   [junit4]   1> Refreshing Kerberos configuration
   [junit4]   1> principal is HTTP/[email protected]
   [junit4]   1> Will use keytab
   [junit4]   1> Commit Succeeded 
   [junit4]   1> 
   [junit4]   2> 
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No 
rules applied to HTTP/[email protected]
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:378)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:348)
   [junit4]   2>        at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]   2>        at javax.security.auth.Subject.doAs(Subject.java:422)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:348)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
   [junit4]   2>        at 
org.apache.solr.security.KerberosFilter.doFilter(KerberosFilter.java:50)
   [junit4]   2>        at 
org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:120)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:253)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:497)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1525783 T9925 oejs.ServletHandler.doHandle WARN 
/vead/p/admin/collections org.apache.solr.common.SolrException: Error during 
request authentication, 
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:261)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:105)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
   [junit4]   2>        at 
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
   [junit4]   2>        at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
   [junit4]   2>        at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2>        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
   [junit4]   2>        at 
org.eclipse.jetty.server.Server.handle(Server.java:497)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
   [junit4]   2>        at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
   [junit4]   2>        at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
   [junit4]   2>        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: 
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No 
rules applied to HTTP/[email protected]
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:378)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:348)
   [junit4]   2>        at java.security.AccessController.doPrivileged(Native 
Method)
   [junit4]   2>        at javax.security.auth.Subject.doAs(Subject.java:422)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:348)
   [junit4]   2>        at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
   [junit4]   2>        at 
org.apache.solr.security.KerberosFilter.doFilter(KerberosFilter.java:50)
   [junit4]   2>        at 
org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:120)
   [junit4]   2>        at 
org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:253)
   [junit4]   2>        ... 23 more
   [junit4]   2> 
   [junit4]   2> 1525790 T9898 oas.SolrTestCaseJ4.tearDown ###Ending 
testKerberizedSolr
   [junit4]   2> 1525860 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ZkController.register We are 
http://127.0.0.1:49735/vead/p/collection1/ and leader is 
http://127.0.0.1:49735/vead/p/collection1/
   [junit4]   2> 1525860 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ZkController.register No LogReplay needed for 
core=collection1 baseURL=http://127.0.0.1:49735/vead/p
   [junit4]   2> 1525860 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ZkController.checkRecovery I am the leader, no 
recovery necessary
   [junit4]   2> 1525861 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ZkController.publish publishing core=collection1 
state=active collection=control_collection
   [junit4]   2> 1525861 T9947 n:127.0.0.1:49735_vead%2Fp c:control_collection 
s:shard1 x:collection1 oasc.ZkController.publish numShards not found on 
descriptor - reading it from system property
   [junit4]   2> 1525862 T9937 n:127.0.0.1:49735_vead%2Fp 
oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path 
/overseer/queue state SyncConnected
   [junit4]   2> 1525863 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"core_node1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "node_name":"127.0.0.1:49735_vead%2Fp",
   [junit4]   2>          "state":"active",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"} current state version: 2
   [junit4]   2> 1525864 T9939 n:127.0.0.1:49735_vead%2Fp 
oasco.ReplicaMutator.updateState Update state numShards=null message={
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"core_node1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "node_name":"127.0.0.1:49735_vead%2Fp",
   [junit4]   2>          "state":"active",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 1526100 T9898 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 
49735
   [junit4]   2> 1526100 T9898 oasc.CoreContainer.shutdown Shutting down 
CoreContainer instance=1440517004
   [junit4]   2> 1526100 T9898 c:control_collection x:collection1 
oasc.ZkController.publish publishing core=collection1 state=down 
collection=control_collection
   [junit4]   2> 1526101 T9898 c:control_collection x:collection1 
oasc.ZkController.publish numShards not found on descriptor - reading it from 
system property
   [junit4]   2> 1526101 T9937 n:127.0.0.1:49735_vead%2Fp 
oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path 
/overseer/queue state SyncConnected
   [junit4]   2> 1526101 T9898 c:control_collection s:shard1 x:collection1 
oasc.SolrCore.close [collection1]  CLOSING SolrCore 
org.apache.solr.core.SolrCore@7cece221
   [junit4]   2> 1526102 T9898 c:control_collection s:shard1 x:collection1 
oasc.ZkController.unregisterConfListener removed listener for config directory 
[/configs/conf1]
   [junit4]   2> 1526102 T9898 c:control_collection s:shard1 x:collection1 
oasc.ZkController.unregisterConfListener No more listeners for config directory 
[/configs/conf1]
   [junit4]   2> 1526102 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"core_node1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "node_name":"127.0.0.1:49735_vead%2Fp",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"} current state version: 3
   [junit4]   2> 1526102 T9898 c:control_collection s:shard1 x:collection1 
oasu.DirectUpdateHandler2.close closing 
DirectUpdateHandler2{commits=0,autocommits=0,soft 
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 1526103 T9898 c:control_collection s:shard1 x:collection1 
oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1526103 T9898 c:control_collection s:shard1 x:collection1 
oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 
0 - closing IndexWriter
   [junit4]   2> 1526103 T9898 c:control_collection s:shard1 x:collection1 
oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with 
IndexWriterCloser
   [junit4]   2> 1526103 T9939 n:127.0.0.1:49735_vead%2Fp 
oasco.ReplicaMutator.updateState Update state numShards=null message={
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "core_node_name":"core_node1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:49735/vead/p";,
   [junit4]   2>          "node_name":"127.0.0.1:49735_vead%2Fp",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":"shard1",
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 1526104 T9898 c:control_collection s:shard1 x:collection1 
oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1526112 T9898 c:control_collection s:shard1 x:collection1 
oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories 
currently being tracked
   [junit4]   2> 1526113 T9898 c:control_collection s:shard1 x:collection1 
oasc.CachingDirectoryFactory.closeCacheValue looking to close 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index 
[CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index;done=false>>]
   [junit4]   2> 1526113 T9898 c:control_collection s:shard1 x:collection1 
oasc.CachingDirectoryFactory.close Closing directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index
   [junit4]   2> 1526113 T9898 c:control_collection s:shard1 x:collection1 
oasc.CachingDirectoryFactory.closeCacheValue looking to close 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data 
[CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data;done=false>>]
   [junit4]   2> 1526114 T9898 c:control_collection s:shard1 x:collection1 
oasc.CachingDirectoryFactory.close Closing directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001/control-001/cores/collection1/data
   [junit4]   2> 1526114 T9898 c:control_collection s:shard1 x:collection1 
oasc.Overseer.close Overseer 
(id=93889084884975620-127.0.0.1:49735_vead%2Fp-n_0000000000) closing
   [junit4]   2> 1526114 T9939 n:127.0.0.1:49735_vead%2Fp 
oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 
127.0.0.1:49735_vead%2Fp
   [junit4]   2> 1526117 T9937 n:127.0.0.1:49735_vead%2Fp 
oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot 
talk to ZK
   [junit4]   2> 1526120 T9898 oejs.AbstractConnector.doStop Stopped 
ServerConnector@350f28a6{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 1526120 T9898 oejsh.ContextHandler.doStop Stopped 
o.e.j.s.ServletContextHandler@795d0f8a{/vead/p,null,UNAVAILABLE}
   [junit4]   2> 1526121 T9898 c:control_collection s:shard1 x:collection1 
oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50190 50190
   [junit4]   2> 1526293 T9906 oasc.ZkTestServer.send4LetterWord connecting to 
127.0.0.1:50190 50190
   [junit4]   2> 1526294 T9906 oasc.ZkTestServer$ZKServerMain.runFromConfig 
WARN Watch limit violations: 
   [junit4]   2>        Maximum concurrent create/delete watches above limit:
   [junit4]   2>        
   [junit4]   2>                3       /solr/aliases.json
   [junit4]   2>                3       /solr/clusterstate.json
   [junit4]   2>        
   [junit4]   2>        Maximum concurrent children watches above limit:
   [junit4]   2>        
   [junit4]   2>                3       /solr/live_nodes
   [junit4]   2>        
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSolrCloudWithKerberos -Dtests.method=testKerberizedSolr 
-Dtests.seed=43F7A82DEDFEC004 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=da -Dtests.timezone=Pacific/Fiji -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   5.50s J1 | TestSolrCloudWithKerberos.testKerberizedSolr <<<
   [junit4]    > Throwable #1: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:49735/vead/p]
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([43F7A82DEDFEC004:E8EE7EEAA79F0033]:0)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1086)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:856)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:799)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
   [junit4]    >        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:152)
   [junit4]    >        at 
org.apache.solr.cloud.TestSolrCloudWithKerberos.testKerberizedSolr(TestSolrCloudWithKerberos.java:165)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:49735/vead/p: Expected mime type 
application/octet-stream but got text/html. <html>
   [junit4]    > <head>
   [junit4]    > <meta http-equiv="Content-Type" 
content="text/html;charset=ISO-8859-1"/>
   [junit4]    > <title>Error 500 </title>
   [junit4]    > </head>
   [junit4]    > <body>
   [junit4]    > <h2>HTTP ERROR: 500</h2>
   [junit4]    > <p>Problem accessing /vead/p/admin/collections. Reason:
   [junit4]    > <pre>    org.apache.solr.common.SolrException: Error during 
request authentication, </pre></p>
   [junit4]    > <hr /><i><small>Powered by Jetty://</small></i>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:529)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:376)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:328)
   [junit4]    >        ... 47 more
   [junit4]   2> 1526302 T9898 c:control_collection s:shard1 x:collection1 
oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J1/temp/solr.cloud.TestSolrCloudWithKerberos
 43F7A82DEDFEC004-001
   [junit4]   2> 5512 T9897 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will 
linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, 
docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, 
locale=da, timezone=Pacific/Fiji
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_25 (64-bit)/cpus=4,threads=1,free=197919392,total=517472256
   [junit4]   2> NOTE: All tests run in this JVM: [AnalysisAfterCoreReloadTest, 
TestFastLRUCache, TestFoldingMultitermQuery, 
ConcurrentDeleteAndCreateCollectionTest, TestShardHandlerFactory, TestRTGBase, 
TestExceedMaxTermLength, CollectionReloadTest, TestCoreDiscovery, 
WordBreakSolrSpellCheckerTest, FieldMutatingUpdateProcessorTest, 
TestReplicaProperties, ZkSolrClientTest, TestSchemaVersionResource, 
RecoveryAfterSoftCommitTest, TestConfigOverlay, BasicFunctionalityTest, 
TestCloudManagedSchema, StatelessScriptUpdateProcessorFactoryTest, 
TestManagedResource, TestComponentsName, TestChildDocTransformer, 
TestLRUStatsCache, SpellingQueryConverterTest, TestDistribDocBasedVersion, 
DistributedTermsComponentTest, TestValueSourceCache, TestStressVersions, 
TestSimpleTrackingShardHandler, SliceStateTest, DistributedQueueTest, 
OverseerStatusTest, CurrencyFieldOpenExchangeTest, TestRebalanceLeaders, 
TestAnalyzeInfixSuggestions, TestCloudPivotFacet, TestHdfsUpdateLog, ZkCLITest, 
DistribCursorPagingTest, TestAtomicUpdateErrorCases, JsonLoaderTest, 
RequestLoggingTest, DistributedSuggestComponentTest, CacheHeaderTest, 
CloudExitableDirectoryReaderTest, MigrateRouteKeyTest, 
BaseCdcrDistributedZkTest, ShardRoutingTest, TestSolrDeletionPolicy2, 
BadComponentTest, TestBulkSchemaConcurrent, TestOmitPositions, 
IndexSchemaRuntimeFieldTest, TestRandomMergePolicy, ShardRoutingCustomTest, 
TriLevelCompositeIdRoutingTest, AnalyticsMergeStrategyTest, 
TestComplexPhraseQParserPlugin, CollectionsAPIAsyncDistributedZkTest, 
ZkStateWriterTest, TestStressUserVersions, TestConfigSets, AddBlockUpdateTest, 
InfoHandlerTest, UpdateParamsTest, TestSolrDynamicMBean, DocumentBuilderTest, 
SolrCloudExampleTest, ReturnFieldsTest, LeaderInitiatedRecoveryOnCommitTest, 
TestDefaultStatsCache, CoreMergeIndexesAdminHandlerTest, 
SolrCmdDistributorTest, ShowFileRequestHandlerTest, EchoParamsTest, 
TestCollationFieldDocValues, TestHashPartitioner, TestIndexSearcher, 
DistanceFunctionTest, ResourceLoaderTest, TestLRUCache, ZkControllerTest, 
CollectionsAPIDistributedZkTest, AutoCommitTest, TestFaceting, TestRecovery, 
DistributedSpellCheckComponentTest, TestRealTimeGet, TestJoin, 
TestSolr4Spatial, SpellCheckComponentTest, PeerSyncTest, BadIndexSchemaTest, 
ConvertedLegacyTest, TestFunctionQuery, OverseerCollectionProcessorTest, 
SoftAutoCommitTest, TestPseudoReturnFields, DefaultValueUpdateProcessorTest, 
DocValuesMultiTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, 
LukeRequestHandlerTest, DirectSolrSpellCheckerTest, 
DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, 
RequestHandlersTest, MoreLikeThisHandlerTest, TestSolrQueryParser, 
IndexSchemaTest, JSONWriterTest, TestAnalyzedSuggestions, SearchHandlerTest, 
TestQuerySenderListener, TestQuerySenderNoQuery, ResponseLogComponentTest, 
TestMergePolicyConfig, MultiTermTest, TestSearchPerf, NumericFieldsTest, 
TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, 
TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnectionTest, 
TestLuceneMatchVersion, SpellPossibilityIteratorTest, TestCharFilters, 
TestCodecSupport, TestLMDirichletSimilarityFactory, 
TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, 
TestBM25SimilarityFactory, TestFastWriter, TestJmxMonitoredMap, 
TimeZoneUtilsTest, TestFastOutputStream, ScriptEngineTest, 
OpenExchangeRatesOrgProviderTest, PluginInfoTest, PreAnalyzedFieldTest, 
TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, 
DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, ClusterStateTest, 
TestUtils, TestDocumentBuilder, ZkNodePropsTest, SystemInfoHandlerTest, 
FileUtilsTest, DistributedMLTComponentTest, CursorPagingTest, 
TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, 
TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ActionThrottleTest, 
AliasIntegrationTest, DistributedVersionInfoTest, 
OutOfBoxZkACLAndCredentialsProvidersTest, OverseerRolesTest, 
ReplicationFactorTest, RollingRestartTest, SaslZkACLProviderTest, 
SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, 
SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestAuthenticationFramework, 
TestCollectionAPI, TestSolrCloudWithKerberos]
   [junit4] Completed [406/502] on J1 in 5.78s, 1 test, 1 error <<< FAILURES!

[...truncated 35 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/init-core-data-001
   [junit4]   2> 1352413 T9029 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl 
(false) and clientAuth (false)
   [junit4]   2> 1352414 T9029 
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system 
property: /bz_ty/uf
   [junit4]   2> 1352417 T9029 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1352417 T9030 oasc.ZkTestServer$2$1.setClientPort client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1352417 T9030 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 1352517 T9029 oasc.ZkTestServer.run start zk server on 
port:48972
   [junit4]   2> 1352518 T9029 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1352519 T9029 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1352521 T9037 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@156c9911 
name:ZooKeeperConnection Watcher:127.0.0.1:48972 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1352521 T9029 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1352522 T9029 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1352522 T9029 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1352524 T9029 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1352528 T9029 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1352535 T9040 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@5de9d5e2 
name:ZooKeeperConnection Watcher:127.0.0.1:48972/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1352535 T9029 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1352536 T9029 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1352536 T9029 oascc.SolrZkClient.makePath makePath: 
/collections/collection1
   [junit4]   2> 1352538 T9029 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/shards
   [junit4]   2> 1352539 T9029 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection
   [junit4]   2> 1352541 T9029 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection/shards
   [junit4]   2> 1352542 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1352542 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.xml
   [junit4]   2> 1352545 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1352545 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/schema.xml
   [junit4]   2> 1352547 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1352547 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1352549 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1352549 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/stopwords.txt
   [junit4]   2> 1352551 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1352551 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/protwords.txt
   [junit4]   2> 1352552 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1352553 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/currency.xml
   [junit4]   2> 1352554 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1352554 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/enumsConfig.xml
   [junit4]   2> 1352556 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1352556 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/open-exchange-rates.json
   [junit4]   2> 1352558 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1352558 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1352560 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1352560 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/old_synonyms.txt
   [junit4]   2> 1352561 T9029 oasc.AbstractZkTestCase.putConfig put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1352562 T9029 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/synonyms.txt
   [junit4]   2> 1352867 T9029 oas.SolrTestCaseJ4.writeCoreProperties Writing 
core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1
   [junit4]   2> 1352869 T9029 oejs.Server.doStart jetty-9.2.10.v20150310
   [junit4]   2> 1352870 T9029 oejsh.ContextHandler.doStart Started 
o.e.j.s.ServletContextHandler@71165cb6{/bz_ty/uf,null,AVAILABLE}
   [junit4]   2> 1352871 T9029 oejs.AbstractConnector.doStart Started 
ServerConnector@4c5ffef0{HTTP/1.1}{127.0.0.1:55426}
   [junit4]   2> 1352871 T9029 oejs.Server.doStart Started @1354526ms
   [junit4]   2> 1352872 T9029 oascse.JettySolrRunner$1.lifeCycleStarted Jetty 
properties: {hostContext=/bz_ty/uf, hostPort=55426, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores}
   [junit4]   2> 1352872 T9029 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 1352872 T9029 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/'
   [junit4]   2> 1352893 T9029 oasc.SolrXmlConfig.fromFile Loading container 
configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/solr.xml
   [junit4]   2> 1352897 T9029 oasc.CorePropertiesLocator.<init> Config-defined 
core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores
   [junit4]   2> 1352898 T9029 oasc.CoreContainer.<init> New CoreContainer 
1766353086
   [junit4]   2> 1352898 T9029 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/]
   [junit4]   2> 1352898 T9029 oasc.CoreContainer.load loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/lib
   [junit4]   2> 1352899 T9029 oasc.SolrResourceLoader.addToClassLoader WARN 
Can't find (or read) directory to add to classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/lib).
   [junit4]   2> 1352905 T9029 oashc.HttpShardHandlerFactory.init created with 
socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 1352906 T9029 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1352907 T9029 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1352907 T9029 oasl.LogWatcher.newRegisteredLogWatcher 
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1352907 T9029 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1352907 T9029 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:48972/solr
   [junit4]   2> 1352908 T9029 oasc.ZkController.checkChrootPath zkHost 
includes chroot
   [junit4]   2> 1352908 T9029 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 1352909 T9029 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1352910 T9054 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@79916f58 
name:ZooKeeperConnection Watcher:127.0.0.1:48972 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1352910 T9029 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1352911 T9029 oascc.SolrZkClient.createZkACLProvider Using 
default ZkACLProvider
   [junit4]   2> 1352912 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ConnectionManager.waitForConnected Waiting for client to connect to 
ZooKeeper
   [junit4]   2> 1352920 T9057 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2573df4b 
name:ZooKeeperConnection Watcher:127.0.0.1:48972/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1352920 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1352921 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1352923 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1352924 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1352926 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1352927 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1352929 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1352930 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1352931 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1352932 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.ZkController.createEphemeralLiveNode Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:55426_bz_ty%2Fuf
   [junit4]   2> 1352933 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55426_bz_ty%2Fuf
   [junit4]   2> 1352934 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1352935 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1352936 T9029 n:127.0.0.1:55426_bz_ty%2Fuf oasc.Overseer.close 
Overseer (id=null) closing
   [junit4]   2> 1352937 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 
127.0.0.1:55426_bz_ty%2Fuf
   [junit4]   2> 1352937 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1352938 T9029 n:127.0.0.1:55426_bz_ty%2Fuf oasc.Overseer.start 
Overseer (id=93889073635000323-127.0.0.1:55426_bz_ty%2Fuf-n_0000000000) starting
   [junit4]   2> 1352940 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1352944 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.OverseerAutoReplicaFailoverThread.<init> Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1352945 T9058 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1352949 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 1352949 T9059 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.OverseerCollectionProcessor.run Process current queue of collection 
creations
   [junit4]   2> 1352951 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CoreContainer.initializeAuthenticationPlugin No authentication plugin used.
   [junit4]   2> 1352952 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CoreContainer.intializeAuthorizationPlugin Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 1352953 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CorePropertiesLocator.discover Looking for core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores
   [junit4]   2> 1352953 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, 
config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1, 
collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1/, coreNodeName=, 
dataDir=data/, shard=}
   [junit4]   2> 1352954 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CorePropertiesLocator.discoverUnder Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1/
   [junit4]   2> 1352954 T9029 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1352955 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
c:control_collection x:collection1 oasc.ZkController.publish publishing 
core=collection1 state=down collection=control_collection
   [junit4]   2> 1352955 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
c:control_collection x:collection1 oasc.ZkController.publish numShards not 
found on descriptor - reading it from system property
   [junit4]   2> 1352956 T9057 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path 
/overseer/queue state SyncConnected
   [junit4]   2> 1352956 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1352956 T9058 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:55426/bz_ty/uf";,
   [junit4]   2>          "node_name":"127.0.0.1:55426_bz_ty%2Fuf",
   [junit4]   2>          "numShards":"1",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":null,
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"} current state version: 0
   [junit4]   2> 1352957 T9058 n:127.0.0.1:55426_bz_ty%2Fuf 
oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2>          "core":"collection1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"http://127.0.0.1:55426/bz_ty/uf";,
   [junit4]   2>          "node_name":"127.0.0.1:55426_bz_ty%2Fuf",
   [junit4]   2>          "numShards":"1",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":null,
   [junit4]   2>          "collection":"control_collection",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 1352957 T9058 n:127.0.0.1:55426_bz_ty%2Fuf 
oasco.ClusterStateMutator.createCollection building a new cName: 
control_collection
   [junit4]   2> 1352957 T9058 n:127.0.0.1:55426_bz_ty%2Fuf 
oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1352958 T9057 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ZkStateReader$2.process A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 1)
   [junit4]   2> 1353957 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for 
collection1
   [junit4]   2> 1353957 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.ZkController.createCollectionZkNode Check for collection 
zkNode:control_collection
   [junit4]   2> 1353958 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1353958 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ZkStateReader.readConfigName Load collection config 
from:/collections/control_collection
   [junit4]   2> 1353959 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oascc.ZkStateReader.readConfigName path=/collections/control_collection 
configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1353959 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1/'
   [junit4]   2> 1353970 T9061 n:127.0.0.1:55426_bz_ty%2Fuf oasc.Config.<init> 
loaded config solrconfig.xml with version 0 
   [junit4]   2> 1353974 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1353978 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1353986 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1353987 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1353991 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1354109 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider.init Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1354115 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1354116 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1354124 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.FileExchangeRateProvider.reload Reloading exchange rates from file 
currency.xml
   [junit4]   2> 1354126 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.FileExchangeRateProvider.reload Reloading exchange rates from file 
currency.xml
   [junit4]   2> 1354128 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 1354128 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 1354129 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 1354129 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 1354130 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 1354130 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 1354130 T9061 n:127.0.0.1:55426_bz_ty%2Fuf 
oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration 
from collection control_collection
   [junit4]   2> 1354130 T9061 n:127.0.0.1:55426_bz_ty%2Fuf x:collection1 
oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1354130 T9061 n:127.0.0.1:55426_bz_ty%2Fuf x:collection1 
oasc.SolrCore.<init> [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1/], dataDir=[null]
   [junit4]   2> 1354131 T9061 n:127.0.0.1:55426_bz_ty%2Fuf x:collection1 
oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to 
JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ea07d8c
   [junit4]   2> 1354131 T9061 n:127.0.0.1:55426_bz_ty%2Fuf x:collection1 
oasc.CachingDirectoryFactory.get return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1/data
   [junit4]   2> 1354131 T9061 n:127.0.0.1:55426_bz_ty%2Fuf x:collection1 
oasc.SolrCore.getNewIndexDir New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 43F7A82DEDFEC004-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1354132 T9061 n:127.0.0.1:55426_bz_ty%2F

[...truncated too long message...]

r.cloud.MultiThreadedOCPTest 
43F7A82DEDFEC004-001/shard-4-001/cores/ocptest_shardsplit2_shard2_1_replica1/data;done=false>>]
   [junit4]   2> 1540768 T9029 oasc.CachingDirectoryFactory.close Closing 
directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 
43F7A82DEDFEC004-001/shard-4-001/cores/ocptest_shardsplit2_shard2_1_replica1/data
   [junit4]   2> 1540769 T9029 oasc.CachingDirectoryFactory.closeCacheValue 
looking to close 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 
43F7A82DEDFEC004-001/shard-4-001/cores/ocptest_shardsplit2_shard2_1_replica1/data/index
 
[CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 
43F7A82DEDFEC004-001/shard-4-001/cores/ocptest_shardsplit2_shard2_1_replica1/data/index;done=false>>]
   [junit4]   2> 1540769 T9029 oasc.CachingDirectoryFactory.close Closing 
directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build/solr-core/test/J0/temp/solr.cloud.MultiThreadedOCPTest
 
43F7A82DEDFEC004-001/shard-4-001/cores/ocptest_shardsplit2_shard2_1_replica1/data/index
   [junit4]   2> 1540770 T9029 oasc.Overseer.close Overseer 
(id=93889073635000332-127.0.0.1:45198_bz_ty%2Fuf-n_0000000004) closing
   [junit4]   2> 1540771 T9380 n:127.0.0.1:45198_bz_ty%2Fuf 
oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 
127.0.0.1:45198_bz_ty%2Fuf
   [junit4]   2> 1542277 T9361 n:127.0.0.1:45198_bz_ty%2Fuf 
oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot 
talk to ZK
   [junit4]   2> 1542281 T9029 oejs.AbstractConnector.doStop Stopped 
ServerConnector@5fdd2212{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 1542281 T9029 oejsh.ContextHandler.doStop Stopped 
o.e.j.s.ServletContextHandler@3291cdb3{/bz_ty/uf,null,UNAVAILABLE}
   [junit4]   2> 1542283 T9029 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 
55426
   [junit4]   2> 1542284 T9029 oasc.ZkTestServer.send4LetterWord connecting to 
127.0.0.1:48972 48972
   [junit4]   2> 1542445 T9030 oasc.ZkTestServer.send4LetterWord connecting to 
127.0.0.1:48972 48972
   [junit4]   2> 1542447 T9030 oasc.ZkTestServer$ZKServerMain.runFromConfig 
WARN Watch limit violations: 
   [junit4]   2>        Maximum concurrent create/delete watches above limit:
   [junit4]   2>        
   [junit4]   2>                6       /solr/aliases.json
   [junit4]   2>                6       /solr/clusterstate.json
   [junit4]   2>                5       /solr/configs/conf1
   [junit4]   2>                4       
/solr/collections/ocptest_shardsplit2/state.json
   [junit4]   2>                4       /solr/collections/ocptest1/state.json
   [junit4]   2>                4       /solr/collections/ocptest2/state.json
   [junit4]   2>                4       /solr/collections/ocptest4/state.json
   [junit4]   2>                4       /solr/collections/ocptest3/state.json
   [junit4]   2>                4       
/solr/collections/ocptest_shardsplit/state.json
   [junit4]   2>        
   [junit4]   2>        Maximum concurrent children watches above limit:
   [junit4]   2>        
   [junit4]   2>                6       /solr/live_nodes
   [junit4]   2>                5       /solr/overseer/collection-queue-work
   [junit4]   2>                4       /solr/overseer/queue
   [junit4]   2>        
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=MultiThreadedOCPTest -Dtests.method=test 
-Dtests.seed=43F7A82DEDFEC004 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=en_CA -Dtests.timezone=America/Montevideo -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    190s J0 | MultiThreadedOCPTest.test <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=9368, 
name=parallelCoreAdminExecutor-4389-thread-15, state=RUNNABLE, 
group=TGRP-MultiThreadedOCPTest]
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([43F7A82DEDFEC004:CBA397F74302ADFC]:0)
   [junit4]    > Caused by: java.lang.AssertionError: Too many closes on 
SolrCore
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([43F7A82DEDFEC004]:0)
   [junit4]    >        at 
org.apache.solr.core.SolrCore.close(SolrCore.java:1150)
   [junit4]    >        at 
org.apache.solr.common.util.IOUtils.closeQuietly(IOUtils.java:31)
   [junit4]    >        at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:652)
   [junit4]    >        at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:611)
   [junit4]    >        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:628)
   [junit4]    >        at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:213)
   [junit4]    >        at 
org.apache.solr.handler.admin.CoreAdminHandler$ParallelCoreAdminHandlerThread.run(CoreAdminHandler.java:1249)
   [junit4]    >        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1542464 T9029 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=CheapBastard, 
sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=en_CA, 
timezone=America/Montevideo
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_25 (64-bit)/cpus=4,threads=1,free=364341216,total=530579456
   [junit4]   2> NOTE: All tests run in this JVM: 
[UniqFieldsUpdateProcessorFactoryTest, DistribDocExpirationUpdateProcessorTest, 
TestPivotHelperCode, TestCollapseQParserPlugin, SpatialHeatmapFacetsTest, 
TestRandomFaceting, TestSchemaResource, TestPhraseSuggestions, 
PingRequestHandlerTest, BJQParserTest, TestBinaryField, 
HdfsBasicDistributedZk2Test, ResponseHeaderTest, TestIndexingPerformance, 
TestClassNameShortening, ConnectionManagerTest, SimplePostToolTest, 
HdfsThreadLeakTest, ExternalFileFieldSortTest, URLClassifyProcessorTest, 
TermVectorComponentTest, DebugComponentTest, DistributedFacetPivotSmallTest, 
TestBadConfig, DateRangeFieldTest, HighlighterTest, TestRangeQuery, 
SolrIndexConfigTest, BasicZkTest, TestRemoteStreaming, TestExpandComponent, 
TestTolerantSearch, TestArbitraryIndexDir, TestLazyCores, MinimalSchemaTest, 
CloudMLTQParserTest, UUIDFieldTest, TestDynamicLoading, 
SegmentsInfoRequestHandlerTest, TestTrackingShardHandlerFactory, 
StandardRequestHandlerTest, TestCSVLoader, SuggesterFSTTest, 
TestDFRSimilarityFactory, QueryElevationComponentTest, 
UpdateRequestProcessorFactoryTest, TestCryptoKeys, 
PreAnalyzedUpdateProcessorTest, LeaderFailoverAfterPartitionTest, 
TestJmxIntegration, TestWordDelimiterFilterFactory, HighlighterConfigTest, 
UUIDUpdateProcessorFallbackTest, TestJsonFacets, 
TestCloudManagedSchemaConcurrent, TestBlendedInfixSuggestions, 
TestSweetSpotSimilarityFactory, TestSolrIndexConfig, 
ChaosMonkeyNothingIsSafeTest, DeleteInactiveReplicaTest, TestConfig, 
TestHighFrequencyDictionaryFactory, DistributedFacetPivotLargeTest, 
AnalysisErrorHandlingTest, TestFieldResource, SpatialRPTFieldTypeTest, 
RemoteQueryErrorTest, DistributedQueryComponentOptimizationTest, 
SignatureUpdateProcessorFactoryTest, SolrRequestParserTest, 
BasicDistributedZkTest, TermsComponentTest, DateMathParserTest, 
DistributedFacetPivotLongTailTest, TestJsonRequest, 
IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestInfoStreamLogging, 
TestDefaultSearchFieldResource, TestReplicationHandler, TestCloudSchemaless, 
OpenCloseCoreStressTest, LeaderElectionIntegrationTest, TestDistributedSearch, 
RequiredFieldsTest, FastVectorHighlighterTest, TestCollationField, 
MBeansHandlerTest, TestPHPSerializedResponseWriter, TestLFUCache, AssignTest, 
CustomCollectionTest, MultiThreadedOCPTest]
   [junit4] Completed [414/502] on J0 in 190.62s, 1 test, 1 error <<< FAILURES!

[...truncated 268 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:526:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:474:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/build.xml:61: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/extra-targets.xml:39:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/build.xml:230:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/solr/common-build.xml:512:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:1415:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-trunk-Java8/lucene/common-build.xml:973:
 There were test failures: 502 suites, 1984 tests, 2 errors, 35 ignored (25 
assumptions)

Total time: 53 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java8 #12
Archived 4 artifacts
Archive block size is 32768
Received 0 blocks and 297064073 bytes
Compression is 0.0%
Took 51 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to