[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_20) - Build # 11296 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11296/
Java: 32bit/jdk1.8.0_20 -client -XX:+UseG1GC

2 tests failed.
REGRESSION:  
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create 
core [halfcollection_shard1_replica1] Caused by: Could not get shard id for 
core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error 
CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core 
[halfcollection_shard1_replica1] Caused by: Could not get shard id for core: 
halfcollection_shard1_replica1
        at 
__randomizedtesting.SeedInfo.seed([4020B111AB407CB7:C1C63F09DC1F1C8B]:0)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
        at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
        at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:38680, 
https://127.0.0.1:40327, https://127.0.0.1:39215]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[https://127.0.0.1:38680, https://127.0.0.1:40327, 
https://127.0.0.1:39215]
        at 
__randomizedtesting.SeedInfo.seed([4020B111AB407CB7:C1C63F09DC1F1C8B]:0)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
        at 
org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:874)
        at 
org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
        at 
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
        at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117)
        at 
org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107)
        at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
        at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not 
 remove replica : customcollreplicadeletion/a/core_node2
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
        ... 46 more




Build Log:
[...truncated 10739 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest
   [junit4]   2> Creating dataDir: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/init-core-data-001
   [junit4]   2> 14897 T65 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl 
(true) and clientAuth (true)
   [junit4]   2> 14897 T65 oas.BaseDistributedSearchTestCase.initHostContext 
Setting hostContext system property: /
   [junit4]   2> 14903 T65 oas.SolrTestCaseJ4.setUp ###Starting 
testDistribSearch
   [junit4]   2> 14915 T65 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 14918 T66 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 15017 T65 oasc.ZkTestServer.run start zk server on port:43690
   [junit4]   2> 15020 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 15063 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 15150 T72 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@168f7b7 name:ZooKeeperConnection 
Watcher:127.0.0.1:43690 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 15151 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 15151 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 15153 T65 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 15191 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 15193 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 15195 T74 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1bc7e4f name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 15196 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 15197 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 15200 T65 oascc.SolrZkClient.makePath makePath: 
/collections/collection1
   [junit4]   2> 15206 T65 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/shards
   [junit4]   2> 15210 T65 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection
   [junit4]   2> 15213 T65 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection/shards
   [junit4]   2> 15217 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 15218 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.xml
   [junit4]   2> 15224 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 15224 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/schema.xml
   [junit4]   2> 15229 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 15230 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 15235 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 15235 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/stopwords.txt
   [junit4]   2> 15240 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 15240 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/protwords.txt
   [junit4]   2> 15245 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 15245 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/currency.xml
   [junit4]   2> 15250 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 15250 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/enumsConfig.xml
   [junit4]   2> 15257 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 15258 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/open-exchange-rates.json
   [junit4]   2> 15263 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 15264 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 15270 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 15270 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/old_synonyms.txt
   [junit4]   2> 15276 T65 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 15277 T65 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/synonyms.txt
   [junit4]   2> 15290 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 15291 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 15294 T76 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@2c0ddc name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 15295 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 15295 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 15715 T65 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 15919 T65 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 15940 T65 oejs.AbstractConnector.doStart Started 
SslSelectChannelConnector@127.0.0.1:40327
   [junit4]   2> 15976 T65 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 15977 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 15977 T65 oasc.SolrResourceLoader.locateSolrHome using system 
property solr.solr.home: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002
   [junit4]   2> 15977 T65 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/'
   [junit4]   2> 15993 T65 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/solr.xml
   [junit4]   2> 16002 T65 oasc.CorePropertiesLocator.<init> Config-defined 
core root directory: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002
   [junit4]   2> 16003 T65 oasc.CoreContainer.<init> New CoreContainer 30631422
   [junit4]   2> 16004 T65 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/]
   [junit4]   2> 16004 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 90000
   [junit4]   2> 16005 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: 
   [junit4]   2> 16005 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 16005 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 16006 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 16006 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 16006 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 16007 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 16007 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 16009 T65 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 16010 T65 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 16010 T65 oasl.LogWatcher.newRegisteredLogWatcher Registering 
Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 16011 T65 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 16011 T65 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:43690/solr
   [junit4]   2> 16025 T65 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 16026 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 16027 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 16029 T88 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1f5a52e name:ZooKeeperConnection 
Watcher:127.0.0.1:43690 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 16030 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 16030 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 16036 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 16038 T90 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1e08f64 name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 16038 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 16046 T65 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 16054 T65 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-queue-work
   [junit4]   2> 16060 T65 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-running
   [junit4]   2> 16065 T65 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-completed
   [junit4]   2> 16071 T65 oascc.SolrZkClient.makePath makePath: 
/overseer/collection-map-failure
   [junit4]   2> 16085 T65 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 16088 T65 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:40327_
   [junit4]   2> 16093 T65 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:40327_
   [junit4]   2> 16100 T65 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 16104 T65 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/election
   [junit4]   2> 16108 T65 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 16119 T65 oasc.OverseerElectionContext.runLeaderProcess I am 
going to be the leader 127.0.0.1:40327_
   [junit4]   2> 16120 T65 oascc.SolrZkClient.makePath makePath: 
/overseer_elect/leader
   [junit4]   2> 16122 T65 oasc.Overseer.start Overseer 
(id=92486776077877252-127.0.0.1:40327_-n_0000000000) starting
   [junit4]   2> 16138 T65 oascc.SolrZkClient.makePath makePath: 
/overseer/queue-work
   [junit4]   2> 16172 T65 oasc.OverseerAutoReplicaFailoverThread.<init> 
Starting OverseerAutoReplicaFailoverThread 
autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=10000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 16178 T92 oasc.OverseerCollectionProcessor.run Process current 
queue of collection creations
   [junit4]   2> 16179 T65 oascc.SolrZkClient.makePath makePath: 
/clusterstate.json
   [junit4]   2> 16182 T65 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 16185 T65 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 16197 T91 oasc.Overseer$ClusterStateUpdater.run Starting to 
work on the main queue
   [junit4]   2> 16198 T65 oasc.CorePropertiesLocator.discover Looking for core 
definitions underneath 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002
   [junit4]   2> 16210 T65 oasc.CorePropertiesLocator.discoverUnder Found core 
conf in 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/conf/
   [junit4]   2> 16342 T65 oasc.CorePropertiesLocator.discover Found 1 core 
definitions
   [junit4]   2> 16344 T65 oass.SolrDispatchFilter.init 
user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 16344 T65 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 16354 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 16355 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 16357 T96 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1bd38dc name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 16357 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 16357 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 16360 T65 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 16367 T65 oasc.ChaosMonkey.monkeyLog monkey: init - expire 
sessions:false cause connection loss:false
   [junit4]   2> 16368 T65 oasc.AbstractFullDistribZkTestBase.createJettys 
Creating collection1 with stateFormat=2
   [junit4]   2> 16368 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 16369 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 16372 T98 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@15090c2 name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 16372 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 16372 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 16381 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 16392 T91 oasc.Overseer$ClusterStateUpdater.buildCollection 
building a new collection: collection1
   [junit4]   2> 16392 T91 oasc.Overseer$ClusterStateUpdater.createCollection 
Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 16399 T91 oasc.Overseer$ClusterStateUpdater.createCollection 
state version collection1 2
   [junit4]   2> 16406 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to create_collection /collections/collection1/state.json
   [junit4]   2> 16413 T100 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 16413 T99 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 16707 T65 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 1
   [junit4]   2> 16708 T65 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 16712 T65 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 16715 T65 oejs.AbstractConnector.doStart Started 
SslSelectChannelConnector@127.0.0.1:39215
   [junit4]   2> 16717 T65 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 16717 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 16718 T65 oasc.SolrResourceLoader.locateSolrHome using system 
property solr.solr.home: 
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003
   [junit4]   2> 16718 T65 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/'
   [junit4]   2> 16733 T65 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/solr.xml
   [junit4]   2> 16744 T65 oasc.CorePropertiesLocator.<init> Config-defined 
core root directory: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003
   [junit4]   2> 16745 T65 oasc.CoreContainer.<init> New CoreContainer 2287350
   [junit4]   2> 16745 T65 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/]
   [junit4]   2> 16746 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 90000
   [junit4]   2> 16746 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: 
   [junit4]   2> 16746 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 16746 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 16747 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 16747 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 16747 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 16748 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 16748 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 16750 T65 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 16751 T65 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 16751 T65 oasl.LogWatcher.newRegisteredLogWatcher Registering 
Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 16751 T65 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 16752 T65 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:43690/solr
   [junit4]   2> 16752 T65 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 16752 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 16754 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 16755 T111 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@6bc1d3 name:ZooKeeperConnection 
Watcher:127.0.0.1:43690 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 16755 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 16756 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 16758 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 16760 T113 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@1d30c9b name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 16760 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 16776 T65 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 17788 T65 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:39215_
   [junit4]   2> 17790 T65 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:39215_
   [junit4]   2> 17796 T65 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 17801 T65 oasc.CorePropertiesLocator.discover Looking for core 
definitions underneath 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003
   [junit4]   2> 17810 T65 oasc.CorePropertiesLocator.discoverUnder Found core 
conf in 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/conf/
   [junit4]   2> 17986 T65 oasc.CorePropertiesLocator.discover Found 1 core 
definitions
   [junit4]   2> 17987 T65 oass.SolrDispatchFilter.init 
user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 17988 T65 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 18282 T65 oasc.AbstractFullDistribZkTestBase.createJettys 
create jetty 2
   [junit4]   2> 18283 T65 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 18287 T65 oejus.SslContextFactory.doStart Enabled Protocols 
[SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, 
TLSv1.1, TLSv1.2]
   [junit4]   2> 18290 T65 oejs.AbstractConnector.doStart Started 
SslSelectChannelConnector@127.0.0.1:38680
   [junit4]   2> 18294 T65 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init()
   [junit4]   2> 18295 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not 
configured for solr (NoInitialContextEx)
   [junit4]   2> 18295 T65 oasc.SolrResourceLoader.locateSolrHome using system 
property solr.solr.home: 
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004
   [junit4]   2> 18296 T65 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004/'
   [junit4]   2> 18313 T65 oasc.ConfigSolr.fromFile Loading container 
configuration from 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004/solr.xml
   [junit4]   2> 18324 T65 oasc.CorePropertiesLocator.<init> Config-defined 
core root directory: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004
   [junit4]   2> 18325 T65 oasc.CoreContainer.<init> New CoreContainer 28182312
   [junit4]   2> 18325 T65 oasc.CoreContainer.load Loading cores into 
CoreContainer 
[instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004/]
   [junit4]   2> 18326 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
socketTimeout to: 90000
   [junit4]   2> 18327 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
urlScheme to: 
   [junit4]   2> 18327 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
connTimeout to: 15000
   [junit4]   2> 18327 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maxConnectionsPerHost to: 20
   [junit4]   2> 18327 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
corePoolSize to: 0
   [junit4]   2> 18328 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maximumPoolSize to: 2147483647
   [junit4]   2> 18328 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
maxThreadIdleTime to: 5
   [junit4]   2> 18328 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
sizeOfQueue to: -1
   [junit4]   2> 18329 T65 oashc.HttpShardHandlerFactory.getParameter Setting 
fairnessPolicy to: false
   [junit4]   2> 18331 T65 oasu.UpdateShardHandler.<init> Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 18332 T65 oasl.LogWatcher.createWatcher SLF4J impl is 
org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 18333 T65 oasl.LogWatcher.newRegisteredLogWatcher Registering 
Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 18333 T65 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 18334 T65 oasc.ZkContainer.initZooKeeper Zookeeper 
client=127.0.0.1:43690/solr
   [junit4]   2> 18334 T65 oasc.ZkController.checkChrootPath zkHost includes 
chroot
   [junit4]   2> 18334 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 18335 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 18337 T126 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@dec7bb name:ZooKeeperConnection 
Watcher:127.0.0.1:43690 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 18338 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 18339 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 18343 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 18345 T128 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@828cb6 name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 18345 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 18359 T65 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 19371 T65 oasc.ZkController.createEphemeralLiveNode Register 
node as live in ZooKeeper:/live_nodes/127.0.0.1:38680_
   [junit4]   2> 19373 T65 oascc.SolrZkClient.makePath makePath: 
/live_nodes/127.0.0.1:38680_
   [junit4]   2> 19377 T65 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 19380 T65 oasc.CorePropertiesLocator.discover Looking for core 
definitions underneath 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004
   [junit4]   2> 19388 T65 oasc.CorePropertiesLocator.discoverUnder Found core 
conf in 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-004/conf/
   [junit4]   2> 19564 T65 oasc.CorePropertiesLocator.discover Found 1 core 
definitions
   [junit4]   2> 19565 T65 oass.SolrDispatchFilter.init 
user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 19565 T65 oass.SolrDispatchFilter.init 
SolrDispatchFilter.init() done
   [junit4]   2> 19570 T65 oasc.AbstractFullDistribZkTestBase.createCollection 
Creating collection with stateFormat=2: customcollreplicadeletion
   [junit4]   2> 19571 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 
ZkCredentialsProvider
   [junit4]   2> 19572 T65 oascc.ConnectionManager.waitForConnected Waiting for 
client to connect to ZooKeeper
   [junit4]   2> 19574 T132 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@19cc4dd name:ZooKeeperConnection 
Watcher:127.0.0.1:43690/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 19574 T65 oascc.ConnectionManager.waitForConnected Client is 
connected to ZooKeeper
   [junit4]   2> 19574 T65 oascc.SolrZkClient.createZkACLProvider Using default 
ZkACLProvider
   [junit4]   2> 19576 T65 
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 20178 T109 oasha.CollectionsHandler.handleCreateAction 
Creating Collection : 
replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 20181 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/collection-queue-work state: 
SyncConnected type NodeChildrenChanged
   [junit4]   2> 20186 T92 oasc.OverseerCollectionProcessor.run Overseer 
Collection Processor: Get the message 
id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>          "operation":"create",
   [junit4]   2>          "fromApi":"true",
   [junit4]   2>          "name":"customcollreplicadeletion",
   [junit4]   2>          "replicationFactor":"1",
   [junit4]   2>          "numShards":"1",
   [junit4]   2>          "maxShardsPerNode":"5",
   [junit4]   2>          "shards":"a,b",
   [junit4]   2>          "stateFormat":"2",
   [junit4]   2>          "router.name":"implicit"}
   [junit4]   2> 20188 T133 oasc.OverseerCollectionProcessor.processMessage 
WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2>          "operation":"create",
   [junit4]   2>          "fromApi":"true",
   [junit4]   2>          "name":"customcollreplicadeletion",
   [junit4]   2>          "replicationFactor":"1",
   [junit4]   2>          "numShards":"1",
   [junit4]   2>          "maxShardsPerNode":"5",
   [junit4]   2>          "shards":"a,b",
   [junit4]   2>          "stateFormat":"2",
   [junit4]   2>          "router.name":"implicit"}
   [junit4]   2> 20191 T133 oasc.OverseerCollectionProcessor.createConfNode 
Only one config set found in zk - using it:conf1
   [junit4]   2> 20192 T133 oasc.OverseerCollectionProcessor.createConfNode 
creating collections conf node /collections/customcollreplicadeletion 
   [junit4]   2> 20193 T133 oascc.SolrZkClient.makePath makePath: 
/collections/customcollreplicadeletion
   [junit4]   2> 20196 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 20200 T91 oasc.Overseer$ClusterStateUpdater.buildCollection 
building a new collection: customcollreplicadeletion
   [junit4]   2> 20200 T91 oasc.Overseer$ClusterStateUpdater.createCollection 
Create collection customcollreplicadeletion with shards [a, b]
   [junit4]   2> 20200 T91 oasc.Overseer$ClusterStateUpdater.createCollection 
state version customcollreplicadeletion 2
   [junit4]   2> 20203 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to create_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 20204 T114 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 20204 T100 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 20205 T134 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 20205 T99 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 20204 T129 oascc.ZkStateReader$2.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 20297 T133 oasc.OverseerCollectionProcessor.createCollection 
Creating SolrCores for new collection customcollreplicadeletion, shardNames [a, 
b] , replicationFactor : 1
   [junit4]   2> 20297 T133 oasc.OverseerCollectionProcessor.createCollection 
Creating shard customcollreplicadeletion_a_replica1 as part of slice a of 
collection customcollreplicadeletion on 127.0.0.1:40327_
   [junit4]   2> 20302 T133 oasc.OverseerCollectionProcessor.createCollection 
Creating shard customcollreplicadeletion_b_replica1 as part of slice b of 
collection customcollreplicadeletion on 127.0.0.1:39215_
   [junit4]   2> 20484 T85 oasha.CoreAdminHandler.handleCreateAction core 
create command 
qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2
   [junit4]   2> 20484 T107 oasha.CoreAdminHandler.handleCreateAction core 
create command 
qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2
   [junit4]   2> 20491 T107 oasc.ZkController.publish publishing 
core=customcollreplicadeletion_b_replica1 state=down 
collection=customcollreplicadeletion
   [junit4]   2> 20492 T85 oasc.ZkController.publish publishing 
core=customcollreplicadeletion_a_replica1 state=down 
collection=customcollreplicadeletion
   [junit4]   2> 20496 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 20497 T107 oasc.ZkController.preRegister Registering watch for 
external collection customcollreplicadeletion
   [junit4]   2> 20497 T107 oascc.ZkStateReader.addZkWatch addZkWatch 
customcollreplicadeletion
   [junit4]   2> 20498 T85 oasc.ZkController.preRegister Registering watch for 
external collection customcollreplicadeletion
   [junit4]   2> 20498 T107 oascc.ZkStateReader.addZkWatch Updating collection 
state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 20501 T91 oasc.Overseer$ClusterStateUpdater.updateState Update 
state numShards=2 message={
   [junit4]   2>          "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"https://127.0.0.1:39215";,
   [junit4]   2>          "node_name":"127.0.0.1:39215_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":"b",
   [junit4]   2>          "collection":"customcollreplicadeletion",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 20501 T107 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 0 
   [junit4]   2> 20502 T107 oasc.ZkController.waitForCoreNodeName look for our 
core node name
   [junit4]   2> 20511 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 20512 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 20513 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 1 
   [junit4]   2> 20514 T85 oascc.ZkStateReader.addZkWatch addZkWatch 
customcollreplicadeletion
   [junit4]   2> 20514 T85 oascc.ZkStateReader.addZkWatch Updating collection 
state at /collections/customcollreplicadeletion/state.json from ZooKeeper... 
   [junit4]   2> 20517 T85 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 1 
   [junit4]   2> 20517 T85 oasc.ZkController.waitForCoreNodeName look for our 
core node name
   [junit4]   2> 20520 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 1 
   [junit4]   2> 20521 T91 oasc.Overseer$ClusterStateUpdater.updateState Update 
state numShards=2 message={
   [junit4]   2>          "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"https://127.0.0.1:40327";,
   [junit4]   2>          "node_name":"127.0.0.1:40327_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "state":"down",
   [junit4]   2>          "shard":"a",
   [junit4]   2>          "collection":"customcollreplicadeletion",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 20523 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 20525 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 20525 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 20525 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 20526 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 2 
   [junit4]   2> 20528 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 2 
   [junit4]   2> 21502 T107 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for customcollreplicadeletion_b_replica1
   [junit4]   2> 21503 T107 oasc.ZkController.createCollectionZkNode Check for 
collection zkNode:customcollreplicadeletion
   [junit4]   2> 21504 T107 oasc.ZkController.createCollectionZkNode Collection 
zkNode exists
   [junit4]   2> 21504 T107 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/customcollreplicadeletion
   [junit4]   2> 21505 T107 oascc.ZkStateReader.readConfigName 
path=/collections/customcollreplicadeletion configName=conf1 specified config 
exists in ZooKeeper
   [junit4]   2> 21506 T107 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/'
   [junit4]   2> 21517 T85 oasc.ZkController.waitForShardId waiting to find 
shard id in clusterstate for customcollreplicadeletion_a_replica1
   [junit4]   2> 21518 T85 oasc.ZkController.createCollectionZkNode Check for 
collection zkNode:customcollreplicadeletion
   [junit4]   2> 21519 T85 oasc.ZkController.createCollectionZkNode Collection 
zkNode exists
   [junit4]   2> 21519 T85 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/customcollreplicadeletion
   [junit4]   2> 21520 T85 oascc.ZkStateReader.readConfigName 
path=/collections/customcollreplicadeletion configName=conf1 specified config 
exists in ZooKeeper
   [junit4]   2> 21521 T85 oasc.SolrResourceLoader.<init> new 
SolrResourceLoader for directory: 
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/'
   [junit4]   2> 21552 T107 oasc.SolrConfig.<init> Using Lucene MatchVersion: 
6.0.0
   [junit4]   2> 21569 T85 oasc.SolrConfig.<init> Using Lucene MatchVersion: 
6.0.0
   [junit4]   2> 21582 T107 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 21583 T107 oass.IndexSchema.readSchema Reading Solr Schema 
from schema.xml
   [junit4]   2> 21595 T85 oasc.SolrConfig.<init> Loaded SolrConfig: 
solrconfig.xml
   [junit4]   2> 21596 T85 oass.IndexSchema.readSchema Reading Solr Schema from 
schema.xml
   [junit4]   2> 21599 T107 oass.IndexSchema.readSchema 
[customcollreplicadeletion_b_replica1] Schema name=test
   [junit4]   2> 21621 T85 oass.IndexSchema.readSchema 
[customcollreplicadeletion_a_replica1] Schema name=test
   [junit4]   2> 21840 T107 oass.OpenExchangeRatesOrgProvider.init Initialized 
with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 21843 T85 oass.OpenExchangeRatesOrgProvider.init Initialized 
with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 21847 T107 oass.IndexSchema.readSchema default search field in 
schema is text
   [junit4]   2> 21849 T107 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 21850 T85 oass.IndexSchema.readSchema default search field in 
schema is text
   [junit4]   2> 21852 T85 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 21869 T107 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 21873 T107 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 21875 T85 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 21877 T107 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 21878 T107 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 21879 T107 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 21879 T85 oass.FileExchangeRateProvider.reload Reloading 
exchange rates from file currency.xml
   [junit4]   2> 21880 T107 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 21881 T107 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 21881 T107 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 21882 T107 oasc.CoreContainer.create Creating SolrCore 
'customcollreplicadeletion_b_replica1' using configuration from collection 
customcollreplicadeletion
   [junit4]   2> 21882 T107 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 21883 T85 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 21883 T107 oasc.SolrCore.<init> 
[customcollreplicadeletion_b_replica1] Opening new SolrCore at 
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/,
 
dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/data/
   [junit4]   2> 21884 T107 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@711e8c
   [junit4]   2> 21884 T85 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 21884 T85 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 21885 T85 oass.OpenExchangeRatesOrgProvider.reload Reloading 
exchange rates from open-exchange-rates.json
   [junit4]   2> 21886 T85 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key 
IMPORTANT NOTE
   [junit4]   2> 21886 T85 
oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, 
got STRING
   [junit4]   2> 21887 T85 oasc.CoreContainer.create Creating SolrCore 
'customcollreplicadeletion_a_replica1' using configuration from collection 
customcollreplicadeletion
   [junit4]   2> 21887 T85 oasc.SolrCore.initDirectoryFactory 
org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 21888 T85 oasc.SolrCore.<init> 
[customcollreplicadeletion_a_replica1] Opening new SolrCore at 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/,
 
dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/data/
   [junit4]   2> 21889 T85 oasc.JmxMonitoredMap.<init> JMX monitoring is 
enabled. Adding Solr mbeans to JMX Server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@711e8c
   [junit4]   2> 21913 T107 oasc.CachingDirectoryFactory.get return new 
directory for 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/data
   [junit4]   2> 21913 T85 oasc.CachingDirectoryFactory.get return new 
directory for 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/data
   [junit4]   2> 21913 T107 oasc.SolrCore.getNewIndexDir New index directory 
detected: old=null 
new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index/
   [junit4]   2> 21914 T85 oasc.SolrCore.getNewIndexDir New index directory 
detected: old=null 
new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index/
   [junit4]   2> 21914 T107 oasc.SolrCore.initIndex WARN 
[customcollreplicadeletion_b_replica1] Solr index directory 
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 21915 T85 oasc.SolrCore.initIndex WARN 
[customcollreplicadeletion_a_replica1] Solr index directory 
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 21915 T107 oasc.CachingDirectoryFactory.get return new 
directory for 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-003/customcollreplicadeletion_b_replica1/data/index
   [junit4]   2> 21916 T85 oasc.CachingDirectoryFactory.get return new 
directory for 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.DeleteLastCustomShardedReplicaTest-4020B111AB407CB7-001/tempDir-002/customcollreplicadeletion_a_replica1/data/index
   [junit4]   2> 21917 T107 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.8831937069946789]
   [junit4]   2> 21917 T85 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=34, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.8831937069946789]
   [junit4]   2> 21918 T107 oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>                
commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-139ffc)),segFN=segments_1,generation=1}
   [junit4]   2> 21918 T85 oasc.SolrDeletionPolicy.onCommit 
SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>                
commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-595bb)),segFN=segments_1,generation=1}
   [junit4]   2> 21919 T85 oasc.SolrDeletionPolicy.updateCommits newest commit 
generation = 1
   [junit4]   2> 21919 T107 oasc.SolrDeletionPolicy.updateCommits newest commit 
generation = 1
   [junit4]   2> 21928 T85 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 21929 T85 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 21930 T85 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 21930 T85 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21931 T107 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 21931 T85 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21931 T107 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 21932 T85 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 21932 T107 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 21933 T85 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 21933 T107 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21933 T85 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 21934 T85 oasc.SolrCore.loadUpdateProcessorChains no 
updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 21934 T107 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 21935 T107 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 21936 T107 oasup.UpdateRequestProcessorChain.init creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 21936 T107 oasup.UpdateRequestProcessorChain.init inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 21937 T107 oasc.SolrCore.loadUpdateProcessorChains no 
updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 21938 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21939 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21940 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21940 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21942 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21942 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21943 T107 oasc.RequestHandlers.initHandlersFromConfig created 
standard: solr.StandardRequestHandler
   [junit4]   2> 21944 T85 oasc.RequestHandlers.initHandlersFromConfig created 
standard: solr.StandardRequestHandler
   [junit4]   2> 21945 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/get: solr.RealTimeGetHandler
   [junit4]   2> 21946 T107 oasc.RequestHandlers.initHandlersFromConfig adding 
lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 21946 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/get: solr.RealTimeGetHandler
   [junit4]   2> 21946 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/replication: solr.ReplicationHandler
   [junit4]   2> 21947 T85 oasc.RequestHandlers.initHandlersFromConfig adding 
lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 21948 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/replication: solr.ReplicationHandler
   [junit4]   2> 21948 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/update: solr.UpdateRequestHandler
   [junit4]   2> 21949 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 21949 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/update: solr.UpdateRequestHandler
   [junit4]   2> 21950 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 21950 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 21951 T107 oasc.RequestHandlers.initHandlersFromConfig created 
/admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 21952 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 21953 T85 oasc.RequestHandlers.initHandlersFromConfig created 
/admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 21953 T107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21955 T107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21955 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21956 T107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21957 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21957 T107 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21959 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21959 T107 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 21960 T107 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 21960 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21961 T107 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.AlcoholicMergePolicy: 
[AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1564841801, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 21962 T107 oasc.SolrDeletionPolicy.onInit 
SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>                
commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-139ffc)),segFN=segments_1,generation=1}
   [junit4]   2> 21962 T107 oasc.SolrDeletionPolicy.updateCommits newest commit 
generation = 1
   [junit4]   2> 21963 T107 oass.SolrIndexSearcher.<init> Opening 
Searcher@1caecee[customcollreplicadeletion_b_replica1] main
   [junit4]   2> 21963 T85 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 21963 T107 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/customcollreplicadeletion
   [junit4]   2> 21963 T85 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 21964 T107 oascc.ZkStateReader.readConfigName 
path=/collections/customcollreplicadeletion configName=conf1 specified config 
exists in ZooKeeper
   [junit4]   2> 21964 T107 oasr.ManagedResourceStorage.newStorageIO Setting up 
ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 21965 T85 oasu.RandomMergePolicy.<init> RandomMergePolicy 
wrapping class org.apache.lucene.index.AlcoholicMergePolicy: 
[AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1564841801, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.1]
   [junit4]   2> 21966 T85 oasc.SolrDeletionPolicy.onInit 
SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>                
commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-595bb)),segFN=segments_1,generation=1}
   [junit4]   2> 21966 T85 oasc.SolrDeletionPolicy.updateCommits newest commit 
generation = 1
   [junit4]   2> 21967 T85 oass.SolrIndexSearcher.<init> Opening 
Searcher@be1e3[customcollreplicadeletion_a_replica1] main
   [junit4]   2> 21967 T85 oascc.ZkStateReader.readConfigName Load collection 
config from:/collections/customcollreplicadeletion
   [junit4]   2> 21968 T107 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured 
ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 21968 T107 oasr.RestManager.init Initializing RestManager with 
initArgs: {}
   [junit4]   2> 21968 T85 oascc.ZkStateReader.readConfigName 
path=/collections/customcollreplicadeletion configName=conf1 specified config 
exists in ZooKeeper
   [junit4]   2> 21969 T85 oasr.ManagedResourceStorage.newStorageIO Setting up 
ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 21968 T107 oasr.ManagedResourceStorage.load Reading 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 21969 T85 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured 
ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 21970 T85 oasr.RestManager.init Initializing RestManager with 
initArgs: {}
   [junit4]   2> 21970 T107 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found 
for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 21970 T85 oasr.ManagedResourceStorage.load Reading 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 21970 T107 oasr.ManagedResourceStorage.load Loaded null at 
path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 21971 T85 
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found 
for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 21971 T85 oasr.ManagedResourceStorage.load Loaded null at path 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 21971 T107 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/_rest_managed.json
   [junit4]   2> 21972 T85 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/_rest_managed.json
   [junit4]   2> 21973 T107 
oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new 
znode /configs/conf1/_rest_managed.json
   [junit4]   2> 21974 T107 oasr.ManagedResourceStorage$JsonStorage.store Saved 
JSON object to path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 21975 T107 oasr.RestManager.init Initializing 0 registered 
ManagedResources
   [junit4]   2> 21980 T85 oasr.ManagedResourceStorage$JsonStorage.store Saved 
JSON object to path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 21981 T85 oasr.RestManager.init Initializing 0 registered 
ManagedResources
   [junit4]   2> 21985 T137 oasc.SolrCore.registerSearcher 
[customcollreplicadeletion_b_replica1] Registered new searcher 
Searcher@1caecee[customcollreplicadeletion_b_replica1] 
main{UninvertingDirectoryReader()}
   [junit4]   2> 21990 T138 oasc.SolrCore.registerSearcher 
[customcollreplicadeletion_a_replica1] Registered new searcher 
Searcher@be1e3[customcollreplicadeletion_a_replica1] 
main{UninvertingDirectoryReader()}
   [junit4]   2> 21998 T107 oasc.CoreContainer.registerCore registering core: 
customcollreplicadeletion_b_replica1
   [junit4]   2> 21998 T107 oasc.ZkController.register Register replica - 
core:customcollreplicadeletion_b_replica1 address:https://127.0.0.1:39215 
collection:customcollreplicadeletion shard:b
   [junit4]   2> 22004 T85 oasc.CoreContainer.registerCore registering core: 
customcollreplicadeletion_a_replica1
   [junit4]   2> 22005 T85 oasc.ZkController.register Register replica - 
core:customcollreplicadeletion_a_replica1 address:https://127.0.0.1:40327 
collection:customcollreplicadeletion shard:a
   [junit4]   2> 22005 T107 oascc.SolrZkClient.makePath makePath: 
/collections/customcollreplicadeletion/leader_elect/b/election
   [junit4]   2> 22006 T85 oascc.SolrZkClient.makePath makePath: 
/collections/customcollreplicadeletion/leader_elect/a/election
   [junit4]   2> 22012 T107 oasc.ShardLeaderElectionContext.runLeaderProcess 
Running the leader process for shard b
   [junit4]   2> 22012 T85 oasc.ShardLeaderElectionContext.runLeaderProcess 
Running the leader process for shard a
   [junit4]   2> 22014 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 22015 T107 
oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found 
to continue.
   [junit4]   2> 22015 T107 oasc.ShardLeaderElectionContext.runLeaderProcess I 
may be the new leader - try and sync
   [junit4]   2> 22015 T85 
oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found 
to continue.
   [junit4]   2> 22016 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I 
may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2 name=customcollreplicadeletion_b_replica1 
org.apache.solr.core.SolrCore@d11253 
url=https://127.0.0.1:39215/customcollreplicadeletion_b_replica1 
node=127.0.0.1:39215_ C2_STATE=coll:customcollreplicadeletion 
core:customcollreplicadeletion_b_replica1 
props:{core=customcollreplicadeletion_b_replica1, 
base_url=https://127.0.0.1:39215, node_name=127.0.0.1:39215_, state=down}
   [junit4]   2> 22017 T107 C2 P39215 oasc.SyncStrategy.sync Sync replicas to 
https://127.0.0.1:39215/customcollreplicadeletion_b_replica1/
   [junit4]   2> ASYNC  NEW_CORE C3 name=customcollreplicadeletion_a_replica1 
org.apache.solr.core.SolrCore@86f66c 
url=https://127.0.0.1:40327/customcollreplicadeletion_a_replica1 
node=127.0.0.1:40327_ C3_STATE=coll:customcollreplicadeletion 
core:customcollreplicadeletion_a_replica1 
props:{core=customcollreplicadeletion_a_replica1, 
base_url=https://127.0.0.1:40327, node_name=127.0.0.1:40327_, state=down}
   [junit4]   2> 22017 T85 C3 P40327 oasc.SyncStrategy.sync Sync replicas to 
https://127.0.0.1:40327/customcollreplicadeletion_a_replica1/
   [junit4]   2> 22019 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 2 
   [junit4]   2> 22019 T107 C2 P39215 oasc.SyncStrategy.syncReplicas Sync 
Success - now sync replicas to me
   [junit4]   2> 22021 T85 C3 P40327 oasc.SyncStrategy.syncReplicas Sync 
Success - now sync replicas to me
   [junit4]   2> 22022 T107 C2 P39215 oasc.SyncStrategy.syncToMe 
https://127.0.0.1:39215/customcollreplicadeletion_b_replica1/ has no replicas
   [junit4]   2> 22022 T85 C3 P40327 oasc.SyncStrategy.syncToMe 
https://127.0.0.1:40327/customcollreplicadeletion_a_replica1/ has no replicas
   [junit4]   2> 22023 T107 oasc.ShardLeaderElectionContext.runLeaderProcess I 
am the new leader: 
https://127.0.0.1:39215/customcollreplicadeletion_b_replica1/ b
   [junit4]   2> 22024 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I 
am the new leader: 
https://127.0.0.1:40327/customcollreplicadeletion_a_replica1/ a
   [junit4]   2> 22025 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 22026 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22026 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22028 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 3 
   [junit4]   2> 22029 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 3 
   [junit4]   2> 22032 T85 oascc.SolrZkClient.makePath makePath: 
/collections/customcollreplicadeletion/leaders/a
   [junit4]   2> 22032 T107 oascc.SolrZkClient.makePath makePath: 
/collections/customcollreplicadeletion/leaders/b
   [junit4]   2> 22043 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 3 
   [junit4]   2> 22045 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 22047 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 22048 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22048 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22049 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 4 
   [junit4]   2> 22051 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 4 
   [junit4]   2> 22054 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 4 
   [junit4]   2> 22058 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 22058 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 22059 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22059 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22060 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 5 
   [junit4]   2> 22062 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 5 
   [junit4]   2> 22064 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 5 
   [junit4]   2> 22067 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 22068 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 22069 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22069 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22072 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 6 
   [junit4]   2> 22073 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 22097 T85 oasc.ZkController.register We are 
https://127.0.0.1:40327/customcollreplicadeletion_a_replica1/ and leader is 
https://127.0.0.1:40327/customcollreplicadeletion_a_replica1/
   [junit4]   2> 22097 T107 oasc.ZkController.register We are 
https://127.0.0.1:39215/customcollreplicadeletion_b_replica1/ and leader is 
https://127.0.0.1:39215/customcollreplicadeletion_b_replica1/
   [junit4]   2> 22098 T85 oasc.ZkController.register No LogReplay needed for 
core=customcollreplicadeletion_a_replica1 baseURL=https://127.0.0.1:40327
   [junit4]   2> 22098 T107 oasc.ZkController.register No LogReplay needed for 
core=customcollreplicadeletion_b_replica1 baseURL=https://127.0.0.1:39215
   [junit4]   2> 22099 T107 oasc.ZkController.checkRecovery I am the leader, no 
recovery necessary
   [junit4]   2> 22099 T107 oasc.ZkController.publish publishing 
core=customcollreplicadeletion_b_replica1 state=active 
collection=customcollreplicadeletion
   [junit4]   2> 22099 T85 oasc.ZkController.checkRecovery I am the leader, no 
recovery necessary
   [junit4]   2> 22100 T85 oasc.ZkController.publish publishing 
core=customcollreplicadeletion_a_replica1 state=active 
collection=customcollreplicadeletion
   [junit4]   2> 22101 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 22104 T107 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 6 
   [junit4]   2> 22104 T85 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 6 
   [junit4]   2> 22107 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 6 
   [junit4]   2> 22108 T91 oasc.Overseer$ClusterStateUpdater.updateState Update 
state numShards=2 message={
   [junit4]   2>          "core":"customcollreplicadeletion_b_replica1",
   [junit4]   2>          "core_node_name":"core_node1",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"https://127.0.0.1:39215";,
   [junit4]   2>          "node_name":"127.0.0.1:39215_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "state":"active",
   [junit4]   2>          "shard":"b",
   [junit4]   2>          "collection":"customcollreplicadeletion",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 22110 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 22111 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22111 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22112 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 7 
   [junit4]   2> 22113 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 7 
   [junit4]   2> 22113 T107 oass.SolrDispatchFilter.handleAdminRequest [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_b_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=b&wt=javabin&version=2}
 status=0 QTime=1631 
   [junit4]   2> 22113 T85 oass.SolrDispatchFilter.handleAdminRequest [admin] 
webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=customcollreplicadeletion_a_replica1&action=CREATE&numShards=2&collection=customcollreplicadeletion&shard=a&wt=javabin&version=2}
 status=0 QTime=1632 
   [junit4]   2> 22117 T91 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 7 
   [junit4]   2> 22117 T91 oasc.Overseer$ClusterStateUpdater.updateState Update 
state numShards=2 message={
   [junit4]   2>          "core":"customcollreplicadeletion_a_replica1",
   [junit4]   2>          "core_node_name":"core_node2",
   [junit4]   2>          "roles":null,
   [junit4]   2>          "base_url":"https://127.0.0.1:40327";,
   [junit4]   2>          "node_name":"127.0.0.1:40327_",
   [junit4]   2>          "numShards":"2",
   [junit4]   2>          "state":"active",
   [junit4]   2>          "shard":"a",
   [junit4]   2>          "collection":"customcollreplicadeletion",
   [junit4]   2>          "operation":"state"}
   [junit4]   2> 22119 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type 
NodeChildrenChanged
   [junit4]   2> 22119 T91 oasc.Overseer$ClusterStateUpdater.updateZkStates 
going to update_collection /collections/customcollreplicadeletion/state.json
   [junit4]   2> 22120 T114 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22120 T99 oascc.ZkStateReader$7.process A cluster state 
change: WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/customcollreplicadeletion/state.json, has occurred - 
updating... 
   [junit4]   2> 22120 T114 oascc.ZkStateReader.updateWatchedCollection 
Updating data for customcollreplicadeletion to ver 8 
   [junit4]   2> 22122 T99 oascc.ZkStateReader.updateWatchedCollection Updating 
data for customcollreplicadeletion to ver 8 
   [junit4]   2> 22162 T133 oasc.OverseerCollectionProcessor.createCollection 
Finished create command on all shards for collection: customcollreplicadeletion
   [junit4]   2> 22176 T133 oasc.OverseerCollectionProcessor$Runner.run 
Overseer Collection Processor: Message 
id:/overseer/collection-queue-work/qn-0000000000 complete, 
response:{success={null={responseHeader={status=0,QTime=1632},core=customcollreplicadeletion_a_replica1},null={responseHeader={status=0,QTime=1631},core=customcollreplicadeletion_b_replica1}}}
   [junit4]   2> 22177 T114 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/collection-queue-work/qnr-0000000000 
state: SyncConnected type NodeDataChanged
   [junit4]   2> 22178 T99 oasc.DistributedQueue$LatchChildWatcher.process 
LatchChildWatcher fired on path: /overseer/collection-queue-work state: 
SyncConnected type NodeChildrenChanged
   [junit4]   2> 22181 T109 oass.SolrDispatchFilter.handleAdminRequest [admin] 
webapp=null path=/admin/collections 
params={replicationFactor=1&shards=a,b&maxShardsPerNode=5&router.name=implicit&name=customcollreplicadeletion&action=CREATE&numShards=1&stateFormat=2&wt=javabin&version=2}
 status=0 QTime=2004 
   [junit4]   2> 22183 T65 
oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to 
finish - collection: customcollreplicadeletion failOnTimeout:true timeout 
(sec):330
   [junit4]   2> 22185 T65 
oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - 
collection: customcollreplicadeletion
   [junit4]   2> 22187 T65 
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default 

[...truncated too long message...]

/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-4020B111AB407CB7-001/tempDir-006/nodes_used_collection_shard1_replica1/data/index
   [junit4]   2> 2115750 T4368 oasc.Overseer.close Overseer 
(id=92486866659115021-127.0.0.1:52481_e_%2Fgy-n_0000000004) closing
   [junit4]   2> 2115750 T4939 oasc.Overseer$ClusterStateUpdater.run Overseer 
Loop exiting : 127.0.0.1:52481_e_%2Fgy
   [junit4]   2> 2117251 T4942 
oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read 
the data org.apache.zookeeper.KeeperException$SessionExpiredException: 
KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2>        at 
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:347)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:344)
   [junit4]   2>        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:344)
   [junit4]   2>        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:383)
   [junit4]   2>        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:126)
   [junit4]   2>        at 
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:333)
   [junit4]   2> 
   [junit4]   2> 2117251 T4918 oascc.ZkStateReader$3.process WARN ZooKeeper 
watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2117278 T4368 oejsh.ContextHandler.doStop stopped 
o.e.j.s.ServletContextHandler{/e_/gy,null}
   [junit4]   2> 2122266 T4368 oas.SolrTestCaseJ4.tearDown ###Ending 
testDistribSearch
   [junit4]   2> 2122268 T4368 oasc.ZkTestServer.send4LetterWord connecting to 
127.0.0.1:54111 54111
   [junit4]   2> 2123149 T4554 oasc.ZkTestServer.send4LetterWord connecting to 
127.0.0.1:54111 54111
   [junit4]   2> 2123152 T4554 oasc.ZkTestServer$ZKServerMain.runFromConfig 
WARN Watch limit violations: 
   [junit4]   2>        Maximum concurrent create/delete watches above limit:
   [junit4]   2>        
   [junit4]   2>                5       
/solr/collections/awholynewstresscollection_collection1_0/state.json
   [junit4]   2>                5       
/solr/collections/awholynewstresscollection_collection0_0/state.json
   [junit4]   2>                4       /solr/aliases.json
   [junit4]   2>                4       /solr/clusterstate.json
   [junit4]   2>                2       
/solr/collections/awholynewcollection_2/state.json
   [junit4]   2>        
   [junit4]   2>        Maximum concurrent data watches above limit:
   [junit4]   2>        
   [junit4]   2>                2       
/solr/collections/awholynewstresscollection_collection0_0/leader_elect/shard2/election/92486866659115021-core_node3-n_0000000001
   [junit4]   2>                2       
/solr/collections/awholynewstresscollection_collection1_0/leader_elect/shard3/election/92486866659115019-core_node9-n_0000000000
   [junit4]   2>        
   [junit4]   2>        Maximum concurrent children watches above limit:
   [junit4]   2>        
   [junit4]   2>                5       /solr/overseer/collection-queue-work
   [junit4]   2>                4       /solr/live_nodes
   [junit4]   2>                3       /solr/overseer/queue
   [junit4]   2>        
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch 
-Dtests.seed=4020B111AB407CB7 -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=ja_JP -Dtests.timezone=Europe/Tirane -Dtests.file.encoding=UTF-8
   [junit4] ERROR    726s J0 | 
CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: 
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error 
CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core 
[halfcollection_shard1_replica1] Caused by: Could not get shard id for core: 
halfcollection_shard1_replica1
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([4020B111AB407CB7:C1C63F09DC1F1C8B]:0)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:550)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    >        at 
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    >        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
   [junit4]    >        at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
   [junit4]    >        at 
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2123179 T4368 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest-4020B111AB407CB7-001
   [junit4]   2> NOTE: test params are: codec=Lucene410: 
{other_tl1=Lucene41(blocksize=128), _version_=Lucene41(blocksize=128), 
multiDefault=PostingsFormat(name=Asserting), 
a_t=PostingsFormat(name=Asserting), intDefault=Lucene41(blocksize=128), 
a_i1=PostingsFormat(name=Asserting), id=PostingsFormat(name=Lucene41WithOrds), 
text=PostingsFormat(name=SimpleText), 
range_facet_l=PostingsFormat(name=Asserting), 
timestamp=PostingsFormat(name=Asserting)}, 
docValues:{timestamp=DocValuesFormat(name=Memory)}, 
sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ja_JP, 
timezone=Europe/Tirane
   [junit4]   2> NOTE: Linux 3.13.0-35-generic i386/Oracle Corporation 1.8.0_20 
(32-bit)/cpus=8,threads=1,free=173696168,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: 
[DocumentAnalysisRequestHandlerTest, DirectSolrSpellCheckerTest, 
AnalyticsMergeStrategyTest, TestRecoveryHdfs, MoreLikeThisHandlerTest, 
TestHighlightDedupGrouping, BadCopyFieldTest, TestSolrCoreProperties, 
TestSystemIdResolver, QueryEqualityTest, ConnectionManagerTest, 
TestManagedSchemaFieldResource, TimeZoneUtilsTest, InfoHandlerTest, 
OpenExchangeRatesOrgProviderTest, CoreAdminHandlerTest, AliasIntegrationTest, 
TestConfigSets, PluginInfoTest, TestStandardQParsers, 
CoreAdminRequestStatusTest, TestRandomMergePolicy, 
ChaosMonkeyNothingIsSafeTest, MinimalSchemaTest, 
OverseerCollectionProcessorTest, RecoveryZkTest, TestSchemaNameResource, 
SignatureUpdateProcessorFactoryTest, HdfsLockFactoryTest, 
TriLevelCompositeIdRoutingTest, SolrCmdDistributorTest, HighlighterConfigTest, 
TestLMDirichletSimilarityFactory, DocExpirationUpdateProcessorFactoryTest, 
BasicDistributedZkTest, ExternalFileFieldSortTest, 
DistribDocExpirationUpdateProcessorTest, UnloadDistributedZkTest, 
TestInitParams, FieldMutatingUpdateProcessorTest, TestFoldingMultitermQuery, 
TestCustomSort, XmlUpdateRequestHandlerTest, NotRequiredUniqueKeyTest, 
DeleteReplicaTest, MultiThreadedOCPTest, TestFastOutputStream, 
SpellingQueryConverterTest, ZkControllerTest, TestComplexPhraseQParserPlugin, 
TestDocumentBuilder, TestCollapseQParserPlugin, JSONWriterTest, 
ParsingFieldUpdateProcessorsTest, PrimitiveFieldTypeTest, 
TestArbitraryIndexDir, DocumentBuilderTest, 
TestSolrQueryParserDefaultOperatorResource, TestManagedSchemaFieldTypeResource, 
TestFieldTypeResource, EchoParamsTest, TestSuggestSpellingConverter, 
BlockDirectoryTest, TestCollationFieldDocValues, TestQuerySenderListener, 
SolrCoreCheckLockOnStartupTest, TestRequestStatusCollectionAPI, TestSearchPerf, 
RequiredFieldsTest, TestTrie, TestHashPartitioner, TestAddFieldRealTimeGet, 
SSLMigrationTest, SolrXmlInZkTest, TestDocBasedVersionConstraints, 
TestSolr4Spatial, IndexBasedSpellCheckerTest, ResponseLogComponentTest, 
TestCoreDiscovery, TestInitQParser, TestDistribDocBasedVersion, 
VMParamsZkACLAndCredentialsProvidersTest, IndexSchemaTest, 
TestSolrDeletionPolicy1, TestFileDictionaryLookup, ZkNodePropsTest, 
CurrencyFieldOpenExchangeTest, AtomicUpdatesTest, TestSolrXmlPersistor, 
DistributedSuggestComponentTest, MBeansHandlerTest, TestFastLRUCache, 
TestCodecSupport, TestWordDelimiterFilterFactory, TestReload, 
TestDynamicFieldCollectionResource, TestDistributedSearch, OverseerTest, 
NoCacheHeaderTest, ClusterStateUpdateTest, TestImplicitCoreProperties, 
DistributedFacetPivotLargeTest, TestAnalyzedSuggestions, 
ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J0 in 726.00s, 1 test, 1 error <<< FAILURES!

[...truncated 659 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:491: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:471: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:60: The following 
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The 
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The 
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: 
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1358:
 The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: 
There were test failures: 429 suites, 1772 tests, 2 errors, 45 ignored (23 
assumptions)

Total time: 100 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_20 -client 
-XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org