Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/847/

1 tests failed.
FAILED:  
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR

Error Message:
Captured an uncaught exception in thread: Thread[id=5532, 
name=coreZkRegister-1177-thread-1, state=RUNNABLE, 
group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=5532, name=coreZkRegister-1177-thread-1, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
        at 
__randomizedtesting.SeedInfo.seed([C18643A00F840DE6:1F0C2DDD34266F15]:0)
Caused by: java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([C18643A00F840DE6]:0)
        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:451)
        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:926)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:881)
        at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10073 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/init-core-data-001
   [junit4]   2> 516930 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[C18643A00F840DE6]-worker)
 [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system 
property: /aw_/
   [junit4]   2> 516933 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 516934 INFO  (Thread-3513) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 516934 INFO  (Thread-3513) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 517034 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ZkTestServer start zk server on port:38041
   [junit4]   2> 517034 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 517035 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 517038 INFO  (zkCallback-456-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3765e6c3 
name:ZooKeeperConnection Watcher:127.0.0.1:38041 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 517038 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 517038 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 517038 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 517040 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 517041 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 517042 INFO  (zkCallback-457-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@13ea22fd 
name:ZooKeeperConnection Watcher:127.0.0.1:38041/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 517042 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 517042 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 517042 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 517044 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 517046 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 517048 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 517049 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 517050 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 517052 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 517052 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 517056 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 517056 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 517058 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 517059 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 517061 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 517061 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 517063 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 517063 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 517066 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 517066 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 517069 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 517069 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 517071 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 517071 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: 
/configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 517073 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 517073 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 517075 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 517075 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 517400 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1
   [junit4]   2> 517406 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 517408 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@71e8a701{/aw_,null,AVAILABLE}
   [junit4]   2> 517416 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@19bcf3d2{HTTP/1.1}{127.0.0.1:33540}
   [junit4]   2> 517416 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.Server Started @520205ms
   [junit4]   2> 517416 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/tempDir-001/control/data,
 hostContext=/aw_, hostPort=33540, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores}
   [junit4]   2> 517417 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 517417 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/'
   [junit4]   2> 517417 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr 
(NoInitialContextEx)
   [junit4]   2> 517418 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 517438 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 517438 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 517440 INFO  (zkCallback-458-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@33b2323e 
name:ZooKeeperConnection Watcher:127.0.0.1:38041/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 517440 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 517441 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 517443 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 517443 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/solr.xml
   [junit4]   2> 517449 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores
   [junit4]   2> 517449 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoreContainer New CoreContainer 1877902892
   [junit4]   2> 517449 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/]
   [junit4]   2> 517449 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/lib
   [junit4]   2> 517449 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/lib).
   [junit4]   2> 517456 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 517457 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 517458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 517458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 517458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38041/solr
   [junit4]   2> 517458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 517458 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 517460 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 517464 INFO  (zkCallback-460-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@546ef8c1 
name:ZooKeeperConnection Watcher:127.0.0.1:38041 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 517467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 517468 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 517472 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 517473 INFO  
(zkCallback-461-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@3b215888 
name:ZooKeeperConnection Watcher:127.0.0.1:38041/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 517473 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 517474 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 517476 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-queue-work
   [junit4]   2> 517479 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-running
   [junit4]   2> 517481 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-completed
   [junit4]   2> 517484 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/collection-map-failure
   [junit4]   2> 517486 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 517488 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 517489 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 517490 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 517491 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:33540_aw_
   [junit4]   2> 517492 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:33540_aw_
   [junit4]   2> 517494 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 517498 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/election
   [junit4]   2> 517499 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 517499 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.LeaderElector Joined leadership election 
with path: 
/overseer_elect/election/94835062127853572-127.0.0.1:33540_aw_-n_0000000000
   [junit4]   2> 517500 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:33540_aw_
   [junit4]   2> 517500 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer_elect/leader
   [junit4]   2> 517502 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer Overseer 
(id=94835062127853572-127.0.0.1:33540_aw_-n_0000000000) starting
   [junit4]   2> 517503 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/overseer/queue-work
   [junit4]   2> 517509 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting 
OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 
autoReplicaFailoverWaitAfterExpiration=30000 
autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 517510 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 517510 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader Updating cluster state from 
ZooKeeper... 
   [junit4]   2> 517516 INFO  
(OverseerCollectionConfigSetProcessor-94835062127853572-127.0.0.1:33540_aw_-n_0000000000)
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.OverseerTaskProcessor Process current 
queue of overseer operations
   [junit4]   2> 517516 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 517543 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 517543 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.CoreContainer No authentication plugin 
used.
   [junit4]   2> 517544 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores
   [junit4]   2> 517545 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1,
 collection=control_collection, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 517545 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/
   [junit4]   2> 517545 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 517546 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ZkController publishing state=down
   [junit4]   2> 517547 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ZkController numShards not found on descriptor - reading it from system 
property
   [junit4]   2> 517549 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:33540_aw_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 517549 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ZkController look for our core node name
   [junit4]   2> 517549 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:33540_aw_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 517550 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
control_collection
   [junit4]   2> 517550 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 517552 INFO  
(zkCallback-461-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 517556 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 517556 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:33540_aw_    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 517560 INFO  
(zkCallback-461-thread-2-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518550 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 518550 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 518550 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 518550 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 518551 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.c.ZkStateReader path=/collections/control_collection configName=conf1 
specified config exists in ZooKeeper
   [junit4]   2> 518551 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/'
   [junit4]   2> 518551 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 518551 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 518567 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] o.a.s.c.Config 
loaded config solrconfig.xml with version 0 
   [junit4]   2> 518571 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 518575 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 518583 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] o.a.s.c.Config 
Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 518584 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.IndexSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 518588 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 518679 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 518684 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 518685 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 518693 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 518695 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 518697 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 518698 WARN  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 518698 WARN  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 518698 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from 
open-exchange-rates.json
   [junit4]   2> 518699 WARN  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 518699 WARN  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 518700 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default 
properties: Can't find resource 'configsetprops.json' in classpath or 
'/configs/conf1', 
cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 518700 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection   x:collection1] 
o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from 
collection control_collection
   [junit4]   2> 518700 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 518700 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/],
 dataDir=[null]
   [junit4]   2> 518701 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr 
mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4cdc1872
   [junit4]   2> 518701 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/data
   [junit4]   2> 518701 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore New index directory detected: old=null 
new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/data/index/
   [junit4]   2> 518701 WARN  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/data/index'
 doesn't exist. Creating new index...
   [junit4]   2> 518702 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/data/index
   [junit4]   2> 518702 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: 
minMergeSize=1677721, mergeFactor=8, maxMergeSize=2147483648, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.7124848842333312]
   [junit4]   2> 518702 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@6f3f70b4 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39f018b),segFN=segments_1,generation=1}
   [junit4]   2> 518702 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 518703 INFO  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
Looking for old index directories to cleanup for core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/data/
   [junit4]   2> 518703 WARN  
(OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.DirectoryFactory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/control-001/cores/collection1/data/
 does not point to a valid data directory; skipping clean-up of old index 
directories.
   [junit4]   2> 518706 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "nodistrib"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "dedupe"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "stored_sig"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating 
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting 
DistributedUpdateProcessorFactory into updateRequestProcessorChain 
"distrib-dup-test-chain-implicit"
   [junit4]   2> 518707 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as 
default, creating implicit default
   [junit4]   2> 518707 WARN  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler 
{type = requestHandler,name = /dump,class = DumpRequestHandler,args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 518709 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 518710 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 518711 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 518712 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 518715 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.RequestHandlers Registered paths: 
/admin/mbeans,standard,/dump,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 518715 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Using default statsCache cache: 
org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 518716 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 518716 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= 
defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 
numVersionBuckets=65536
   [junit4]   2> 518716 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Hard AutoCommit: disabled
   [junit4]   2> 518716 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore Soft AutoCommit: disabled
   [junit4]   2> 518717 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=37, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=71.693359375, 
floorSegmentMB=1.576171875, forceMergeDeletesPctAllowed=1.3312590507759703, 
segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.6840759245306348
   [junit4]   2> 518717 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2>        commit{dir=MockDirectoryWrapper(RAMDirectory@6f3f70b4 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39f018b),segFN=segments_1,generation=1}
   [junit4]   2> 518717 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.SolrCore newest commit generation = 1
   [junit4]   2> 518717 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@70e974ff[collection1] 
main
   [junit4]   2> 518718 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader Load collection config 
from:/collections/control_collection
   [junit4]   2> 518718 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.c.ZkStateReader path=/collections/control_collection 
configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 518718 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based 
storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO 
with znodeBase: /configs/conf1
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode 
/configs/conf1/_rest_managed.json
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path 
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 518719 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 518720 INFO  
(searcherExecutor-1140-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Registered new searcher Searcher@70e974ff[collection1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 518720 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed 
version buckets
   [junit4]   2> 518720 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 
65536 version buckets from index
   [junit4]   2> 518720 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed 
version bucket highest value from index
   [junit4]   2> 518720 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent 
updates, using new clock 1517360995917365248
   [junit4]   2> 518722 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with 
highest version 1517360995917365248
   [junit4]   2> 518722 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 518722 INFO  
(coreLoadExecutor-1139-thread-1-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_ c:control_collection s:shard1 r:core_node1 
x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 518723 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
Register replica - core:collection1 address:http://127.0.0.1:33540/aw_ 
collection:control_collection shard:shard1
   [junit4]   2> 518724 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: 
/collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 518728 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector 
Joined leadership election with path: 
/collections/control_collection/leader_elect/shard1/election/94835062127853572-core_node1-n_0000000000
   [junit4]   2> 518729 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Running the leader process for shard shard1
   [junit4]   2> 518730 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 518730 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 518730 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:33540/aw_/collection1/
   [junit4]   2> 518730 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 518730 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy 
http://127.0.0.1:33540/aw_/collection1/ has no replicas
   [junit4]   2> 518730 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:33540/aw_/collection1/ shard1
   [junit4]   2> 518731 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 518731 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 518734 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] 
o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 518736 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:33540_aw_",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "core":"collection1"} current state version: 1
   [junit4]   2> 518737 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ReplicaMutator Update state 
numShards=null message={
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:33540_aw_",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "core":"collection1"}
   [junit4]   2> 518738 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "core":"collection1"} current state version: 1
   [junit4]   2> 518841 INFO  
(zkCallback-461-thread-2-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518841 INFO  
(zkCallback-461-thread-3-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518888 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
We are http://127.0.0.1:33540/aw_/collection1/ and leader is 
http://127.0.0.1:33540/aw_/collection1/
   [junit4]   2> 518888 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33540/aw_
   [junit4]   2> 518888 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 518888 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
publishing state=active
   [junit4]   2> 518888 INFO  
(coreZkRegister-1133-thread-1-processing-n:127.0.0.1:33540_aw_ x:collection1 
s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33540_aw_ 
c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 518890 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:33540_aw_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 518890 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33540/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:33540_aw_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 518993 INFO  
(zkCallback-461-thread-3-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518993 INFO  
(zkCallback-461-thread-2-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 1)
   [junit4]   2> 519059 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 519068 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 519097 INFO  (zkCallback-463-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@4ad8730c 
name:ZooKeeperConnection Watcher:127.0.0.1:38041/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 519097 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 519097 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 519097 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 519099 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause 
connection loss:false
   [junit4]   2> 519422 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores/collection1
   [junit4]   2> 519423 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001
   [junit4]   2> 519424 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.Server jetty-9.2.13.v20150730
   [junit4]   2> 519426 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@4c561ed7{/aw_,null,AVAILABLE}
   [junit4]   2> 519427 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.ServerConnector Started 
ServerConnector@c27b5d2{HTTP/1.1}{127.0.0.1:54285}
   [junit4]   2> 519427 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.Server Started @522216ms
   [junit4]   2> 519427 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/tempDir-001/jetty1,
 solrconfig=solrconfig.xml, hostContext=/aw_, hostPort=54285, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores}
   [junit4]   2> 519427 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): 
sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 519427 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/'
   [junit4]   2> 519428 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr 
(NoInitialContextEx)
   [junit4]   2> 519428 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not 
find system property or JNDI)
   [junit4]   2> 519445 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 519446 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 519448 INFO  (zkCallback-464-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@7b1f15d5 
name:ZooKeeperConnection Watcher:127.0.0.1:38041/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 519448 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 519449 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 519450 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 519450 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/solr.xml
   [junit4]   2> 519455 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoresLocator Config-defined core root directory: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores
   [junit4]   2> 519455 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoreContainer New CoreContainer 853014287
   [junit4]   2> 519455 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer 
[instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/]
   [junit4]   2> 519455 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.CoreContainer loading shared library: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/lib
   [junit4]   2> 519455 WARN  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.SolrResourceLoader Can't find (or read) directory to add to 
classloader: lib (resolved as: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/lib).
   [junit4]   2> 519466 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 
90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 
20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 
2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : 
false,useRetries : false,
   [junit4]   2> 519467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with 
params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 519467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 519467 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j 
(org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 519468 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38041/solr
   [junit4]   2> 519468 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 519468 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 519468 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 519469 INFO  (zkCallback-466-thread-1) [    ] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@32e9f506 
name:ZooKeeperConnection Watcher:127.0.0.1:38041 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 519470 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 519471 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 519472 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.c.ConnectionManager Waiting for client to 
connect to ZooKeeper
   [junit4]   2> 519473 INFO  
(zkCallback-467-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_    ] o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@7be8f674 
name:ZooKeeperConnection Watcher:127.0.0.1:38041/solr got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 519473 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.c.ConnectionManager Client is connected to 
ZooKeeper
   [junit4]   2> 519478 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.c.ZkStateReader Updating cluster state 
from ZooKeeper... 
   [junit4]   2> 520482 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:54285_aw_
   [junit4]   2> 520483 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.c.SolrZkClient makePath: 
/live_nodes/127.0.0.1:54285_aw_
   [junit4]   2> 520484 INFO  
(zkCallback-461-thread-2-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 520484 INFO  
(zkCallback-467-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 520484 INFO  
(zkCallback-461-thread-3-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A live node change: 
WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has 
occurred - updating... (live nodes size: 1)
   [junit4]   2> 520484 INFO  (zkCallback-463-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A live node change: WatchedEvent state:SyncConnected 
type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live 
nodes size: 1)
   [junit4]   2> 520488 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 520488 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.LeaderElector Joined leadership election 
with path: 
/overseer_elect/election/94835062127853576-127.0.0.1:54285_aw_-n_0000000001
   [junit4]   2> 520489 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.LeaderElector Watching path 
/overseer_elect/election/94835062127853572-127.0.0.1:33540_aw_-n_0000000000 to 
know if I could be the leader
   [junit4]   2> 520501 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.CoreContainer Security conf doesn't exist. 
Skipping setup for authorization module.
   [junit4]   2> 520502 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.CoreContainer No authentication plugin 
used.
   [junit4]   2> 520502 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.CoresLocator Looking for core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores
   [junit4]   2> 520503 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.SolrCore Created CoreDescriptor: 
{name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, 
loadOnStartup=true, configSetProperties=configsetprops.json, 
instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores/collection1,
 collection=collection1, 
absoluteInstDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores/collection1/,
 coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 520503 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.CoresLocator Found core collection1 in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores/collection1/
   [junit4]   2> 520503 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.c.CoresLocator Found 1 core definitions
   [junit4]   2> 520504 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.s.SolrDispatchFilter 
user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 520504 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:54285_aw_    ] o.a.s.s.SolrDispatchFilter 
SolrDispatchFilter.init() done
   [junit4]   2> 520504 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.ZkController 
publishing state=down
   [junit4]   2> 520504 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.ZkController 
numShards not found on descriptor - reading it from system property
   [junit4]   2> 520506 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.Overseer processMessage: queueSize: 1, 
message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54285/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:54285_aw_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 520506 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.ZkController look 
for our core node name
   [junit4]   2> 520506 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 
message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54285/aw_";,
   [junit4]   2>   "node_name":"127.0.0.1:54285_aw_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 520506 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ClusterStateMutator building a new cName: 
collection1
   [junit4]   2> 520507 INFO  
(OverseerStateUpdate-94835062127853572-127.0.0.1:33540_aw_-n_0000000000) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.o.ReplicaMutator Assigning new node to 
shard shard=shard1
   [junit4]   2> 520508 INFO  
(zkCallback-467-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520508 INFO  (zkCallback-463-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: WatchedEvent 
state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred 
- updating... (live nodes size: 2)
   [junit4]   2> 520508 INFO  
(zkCallback-461-thread-2-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520508 INFO  
(zkCallback-461-thread-3-processing-n:127.0.0.1:33540_aw_) 
[n:127.0.0.1:33540_aw_    ] o.a.s.c.c.ZkStateReader A cluster state change: 
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, 
has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521507 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.ZkController 
waiting to find shard id in clusterstate for collection1
   [junit4]   2> 521507 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.ZkController 
Check for collection zkNode:collection1
   [junit4]   2> 521507 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.ZkController 
Collection zkNode exists
   [junit4]   2> 521507 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
Load collection config from:/collections/collection1
   [junit4]   2> 521508 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader 
path=/collections/collection1 configName=conf1 specified config exists in 
ZooKeeper
   [junit4]   2> 521508 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] 
o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: 
'/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.LeaderInitiatedRecoveryOnShardRestartTest_C18643A00F840DE6-001/shard-1-001/cores/collection1/'
   [junit4]   2> 521508 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] 
o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 521508 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 521524 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.Config loaded 
config solrconfig.xml with version 0 
   [junit4]   2> 521528 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.SolrConfig 
current version of requestparams : -1
   [junit4]   2> 521531 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using 
Lucene MatchVersion: 6.0.0
   [junit4]   2> 521539 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.c.Config Loaded 
SolrConfig: solrconfig.xml
   [junit4]   2> 521540 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 521555 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
[collection1] Schema name=test
   [junit4]   2> 521635 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] 
o.a.s.s.OpenExchangeRatesOrgProvider Initialized with 
rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 521641 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
default search field in schema is text
   [junit4]   2> 521642 INFO  
(coreLoadExecutor-1150-thread-1-processing-n:127.0.0.1:54285_aw_) 
[n:127.0.0.1:54285_aw_ c:collection1   x:collection1] o.a.s.s.IndexSchema 
unique key field: id
   [junit4]   2> 521653 INFO  (coreLoadExecutor-1150-thread-1-process

[...truncated too long message...]

llections/collection1/leader_elect/shard1/election/94835062127853586-core_node2-n_0000000003
   [junit4]   2> 559661 WARN  
(zkCallback-482-thread-1-processing-n:127.0.0.1:49855_aw_) 
[n:127.0.0.1:49855_aw_ c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.c.LeaderElector 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: 
KeeperErrorCode = Session expired for 
/collections/collection1/leader_elect/shard1/election/94835062127853586-core_node2-n_0000000003
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2>        at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2>        at 
org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:244)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:241)
   [junit4]   2>        at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:241)
   [junit4]   2>        at 
org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:86)
   [junit4]   2>        at 
org.apache.solr.cloud.ShardLeaderElectionContextBase.cancelElection(ElectionContext.java:169)
   [junit4]   2>        at 
org.apache.solr.cloud.ShardLeaderElectionContext.cancelElection(ElectionContext.java:285)
   [junit4]   2>        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:467)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:58)
   [junit4]   2>        at 
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:388)
   [junit4]   2>        at 
org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:269)
   [junit4]   2>        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2>        at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 559662 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.ServerConnector Stopped 
ServerConnector@3032383c{HTTP/1.1}{127.0.0.1:49855}
   [junit4]   2> 559663 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [    ] o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@672189f9{/aw_,null,UNAVAILABLE}
   [junit4]   2> 559665 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:49855_aw_ c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.c.ChaosMonkey monkey: stop shard! 33540
   [junit4]   2> 559665 INFO  
(TEST-LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR-seed#[C18643A00F840DE6])
 [n:127.0.0.1:49855_aw_ c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:38041 38041
   [junit4]   2> 559834 INFO  (Thread-3513) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:38041 38041
   [junit4]   2> 559835 WARN  (Thread-3513) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        7       /solr/aliases.json
   [junit4]   2>        6       /solr/security.json
   [junit4]   2>        5       /solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        34      /solr/clusterstate.json
   [junit4]   2>        3       
/solr/overseer_elect/election/94835062127853572-127.0.0.1:33540_aw_-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        11      /solr/live_nodes
   [junit4]   2>        8       /solr/collections
   [junit4]   2>        2       /solr/overseer/queue
   [junit4]   2>        2       /solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=LeaderInitiatedRecoveryOnShardRestartTest 
-Dtests.method=testRestartWithAllInLIR -Dtests.seed=C18643A00F840DE6 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=es_MX -Dtests.timezone=America/Glace_Bay -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   42.9s J2 | 
LeaderInitiatedRecoveryOnShardRestartTest.testRestartWithAllInLIR <<<
   [junit4]    > Throwable #1: 
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=5532, name=coreZkRegister-1177-thread-1, 
state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([C18643A00F840DE6:1F0C2DDD34266F15]:0)
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([C18643A00F840DE6]:0)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.updateLeaderInitiatedRecoveryState(ZkController.java:2126)
   [junit4]    >        at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:451)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:197)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]    >        at 
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:346)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1113)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:926)
   [junit4]    >        at 
org.apache.solr.cloud.ZkController.register(ZkController.java:881)
   [junit4]    >        at 
org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
   [junit4]    >        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    >        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 559841 INFO  
(SUITE-LeaderInitiatedRecoveryOnShardRestartTest-seed#[C18643A00F840DE6]-worker)
 [n:127.0.0.1:49855_aw_ c:collection1 s:shard1 r:core_node2 x:collection1] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Nov 09, 2015 11:24:11 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60), 
sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_MX, 
timezone=America/Glace_Bay
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 
1.8.0_66 (64-bit)/cpus=4,threads=1,free=256486664,total=479199232
   [junit4]   2> NOTE: All tests run in this JVM: 
[StatelessScriptUpdateProcessorFactoryTest, LukeRequestHandlerTest, 
SharedFSAutoReplicaFailoverTest, TestSchemaResource, CurrencyFieldXmlFileTest, 
TestCursorMarkWithoutUniqueKey, BadComponentTest, OverseerTaskQueueTest, 
TestMiniSolrCloudClusterSSL, AutoCommitTest, 
SharedFSAutoReplicaFailoverUtilsTest, TestSolrConfigHandlerConcurrent, 
SparseHLLTest, SimpleCollectionCreateDeleteTest, RequestLoggingTest, 
TestWriterPerf, BadCopyFieldTest, ConnectionManagerTest, TestBulkSchemaAPI, 
TestIntervalFaceting, RecoveryZkTest, DistributedVersionInfoTest, 
OverriddenZkACLAndCredentialsProvidersTest, ShowFileRequestHandlerTest, 
BinaryUpdateRequestHandlerTest, TestConfigSetsAPI, TestGroupingSearch, 
ScriptEngineTest, RegexBoostProcessorTest, HdfsWriteToMultipleCollectionsTest, 
TestSolrJ, DistributedFacetPivotLongTailTest, 
LeaderInitiatedRecoveryOnShardRestartTest]
   [junit4] Completed [86/555] on J2 in 43.53s, 1 test, 1 error <<< FAILURES!

[...truncated 1521 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:782:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:719:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1432:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:987:
 There were test failures: 555 suites (8 ignored), 2200 tests, 1 error, 32 
ignored (18 assumptions) [seed: C18643A00F840DE6]

Total time: 206 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to