Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11419/
Java: 32bit/jdk1.8.0_40-ea-b04 -client -XX:+UseG1GC
1 tests failed.
REGRESSION:
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
There were too many update fails - we expect it can happen, but shouldn't easily
Stack Trace:
java.lang.AssertionError: There were too many update fails - we expect it can
happen, but shouldn't easily
at
__randomizedtesting.SeedInfo.seed([46AACD3FB0232159:C74C4327C77C4165]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:223)
at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12023 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> Creating dataDir:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/init-core-data-001
[junit4] 2> 2279043 T7735
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system
property: /
[junit4] 2> 2279050 T7735 oas.SolrTestCaseJ4.setUp ###Starting
testDistribSearch
[junit4] 2> 2279050 T7735 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2279051 T7736 oasc.ZkTestServer$ZKServerMain.runFromConfig
Starting server
[junit4] 2> 2279151 T7735 oasc.ZkTestServer.run start zk server on
port:52726
[junit4] 2> 2279152 T7735
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
ZkCredentialsProvider
[junit4] 2> 2279153 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2279158 T7742 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1004dbd name:ZooKeeperConnection
Watcher:127.0.0.1:52726 got event WatchedEvent state:SyncConnected type:None
path:null path:null type:None
[junit4] 2> 2279158 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2279159 T7735 oascc.SolrZkClient.createZkACLProvider Using
default ZkACLProvider
[junit4] 2> 2279159 T7735 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2279165 T7735
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
ZkCredentialsProvider
[junit4] 2> 2279167 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2279169 T7744 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@452495 name:ZooKeeperConnection
Watcher:127.0.0.1:52726/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2279170 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2279171 T7735 oascc.SolrZkClient.createZkACLProvider Using
default ZkACLProvider
[junit4] 2> 2279171 T7735 oascc.SolrZkClient.makePath makePath:
/collections/collection1
[junit4] 2> 2279176 T7735 oascc.SolrZkClient.makePath makePath:
/collections/collection1/shards
[junit4] 2> 2279180 T7735 oascc.SolrZkClient.makePath makePath:
/collections/control_collection
[junit4] 2> 2279183 T7735 oascc.SolrZkClient.makePath makePath:
/collections/control_collection/shards
[junit4] 2> 2279188 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
to /configs/conf1/solrconfig.xml
[junit4] 2> 2279189 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/solrconfig.xml
[junit4] 2> 2279194 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
to /configs/conf1/schema.xml
[junit4] 2> 2279195 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/schema.xml
[junit4] 2> 2279200 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2279200 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2279206 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
to /configs/conf1/stopwords.txt
[junit4] 2> 2279207 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/stopwords.txt
[junit4] 2> 2279213 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
to /configs/conf1/protwords.txt
[junit4] 2> 2279214 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/protwords.txt
[junit4] 2> 2279220 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
to /configs/conf1/currency.xml
[junit4] 2> 2279221 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/currency.xml
[junit4] 2> 2279227 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
to /configs/conf1/enumsConfig.xml
[junit4] 2> 2279228 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/enumsConfig.xml
[junit4] 2> 2279233 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2279234 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/open-exchange-rates.json
[junit4] 2> 2279239 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2279240 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2279245 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
to /configs/conf1/old_synonyms.txt
[junit4] 2> 2279246 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/old_synonyms.txt
[junit4] 2> 2279252 T7735 oasc.AbstractZkTestCase.putConfig put
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
to /configs/conf1/synonyms.txt
[junit4] 2> 2279253 T7735 oascc.SolrZkClient.makePath makePath:
/configs/conf1/synonyms.txt
[junit4] 2> 2279624 T7735 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2279645 T7735 oejs.AbstractConnector.doStart Started
[email protected]:49660
[junit4] 2> 2279646 T7735 oass.SolrDispatchFilter.init
SolrDispatchFilter.init()
[junit4] 2> 2279647 T7735 oasc.SolrResourceLoader.locateSolrHome JNDI not
configured for solr (NoInitialContextEx)
[junit4] 2> 2279647 T7735 oasc.SolrResourceLoader.locateSolrHome using
system property solr.solr.home:
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002
[junit4] 2> 2279648 T7735 oasc.SolrResourceLoader.<init> new
SolrResourceLoader for directory:
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/'
[junit4] 2> 2279672 T7735 oasc.ConfigSolr.fromFile Loading container
configuration from
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/solr.xml
[junit4] 2> 2279689 T7735 oasc.CoreContainer.<init> New CoreContainer
31516525
[junit4] 2> 2279690 T7735 oasc.CoreContainer.load Loading cores into
CoreContainer
[instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/]
[junit4] 2> 2279691 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting socketTimeout to: 90000
[junit4] 2> 2279691 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting urlScheme to:
[junit4] 2> 2279692 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting connTimeout to: 15000
[junit4] 2> 2279692 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnectionsPerHost to: 20
[junit4] 2> 2279692 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnections to: 10000
[junit4] 2> 2279693 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting corePoolSize to: 0
[junit4] 2> 2279693 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maximumPoolSize to: 2147483647
[junit4] 2> 2279693 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxThreadIdleTime to: 5
[junit4] 2> 2279694 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting sizeOfQueue to: -1
[junit4] 2> 2279694 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting fairnessPolicy to: false
[junit4] 2> 2279695 T7735 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params:
socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2279695 T7735 oasl.LogWatcher.createWatcher SLF4J impl is
org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2279696 T7735 oasl.LogWatcher.newRegisteredLogWatcher
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2279696 T7735 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2279696 T7735 oasc.ZkContainer.initZooKeeper Zookeeper
client=127.0.0.1:52726/solr
[junit4] 2> 2279697 T7735 oasc.ZkController.checkChrootPath zkHost
includes chroot
[junit4] 2> 2279697 T7735
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
ZkCredentialsProvider
[junit4] 2> 2279698 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2279700 T7755 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1b8d24c name:ZooKeeperConnection
Watcher:127.0.0.1:52726 got event WatchedEvent state:SyncConnected type:None
path:null path:null type:None
[junit4] 2> 2279700 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2279700 T7735 oascc.SolrZkClient.createZkACLProvider Using
default ZkACLProvider
[junit4] 2> 2279703 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2279705 T7757 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1411196 name:ZooKeeperConnection
Watcher:127.0.0.1:52726/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2279705 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2279707 T7735 oascc.SolrZkClient.makePath makePath:
/overseer/queue
[junit4] 2> 2279712 T7735 oascc.SolrZkClient.makePath makePath:
/overseer/collection-queue-work
[junit4] 2> 2279716 T7735 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-running
[junit4] 2> 2279720 T7735 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-completed
[junit4] 2> 2279724 T7735 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-failure
[junit4] 2> 2279728 T7735 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2279731 T7735 oasc.ZkController.createEphemeralLiveNode
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49660_
[junit4] 2> 2279732 T7735 oascc.SolrZkClient.makePath makePath:
/live_nodes/127.0.0.1:49660_
[junit4] 2> 2279735 T7735 oascc.SolrZkClient.makePath makePath:
/overseer_elect
[junit4] 2> 2279738 T7735 oascc.SolrZkClient.makePath makePath:
/overseer_elect/election
[junit4] 2> 2279740 T7735 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2279742 T7735 oasc.OverseerElectionContext.runLeaderProcess I
am going to be the leader 127.0.0.1:49660_
[junit4] 2> 2279743 T7735 oascc.SolrZkClient.makePath makePath:
/overseer_elect/leader
[junit4] 2> 2279745 T7735 oasc.Overseer.start Overseer
(id=92595381938290691-127.0.0.1:49660_-n_0000000000) starting
[junit4] 2> 2279748 T7735 oascc.SolrZkClient.makePath makePath:
/overseer/queue-work
[junit4] 2> 2279758 T7735 oasc.OverseerAutoReplicaFailoverThread.<init>
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2279759 T7759 oasc.OverseerCollectionProcessor.run Process
current queue of collection creations
[junit4] 2> 2279760 T7735 oascc.SolrZkClient.makePath makePath:
/clusterstate.json
[junit4] 2> 2279766 T7735 oascc.SolrZkClient.makePath makePath:
/aliases.json
[junit4] 2> 2279769 T7735
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2279776 T7758 oasc.Overseer$ClusterStateUpdater.run Starting
to work on the main queue
[junit4] 2> 2279778 T7761 oasc.ZkController.publish publishing
core=collection1 state=down collection=control_collection
[junit4] 2> 2279779 T7761 oasc.ZkController.publish numShards not found on
descriptor - reading it from system property
[junit4] 2> 2279780 T7761 oasc.ZkController.waitForCoreNodeName look for
our core node name
[junit4] 2> 2279781 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2279788 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:49660";,
[junit4] 2> "node_name":"127.0.0.1:49660_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 2279788 T7758
oasc.Overseer$ClusterStateUpdater.createCollection Create collection
control_collection with shards [shard1]
[junit4] 2> 2279788 T7758
oasc.Overseer$ClusterStateUpdater.createCollection state version
control_collection 1
[junit4] 2> 2279789 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard1
[junit4] 2> 2279794 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2280783 T7761 oasc.ZkController.waitForShardId waiting to find
shard id in clusterstate for collection1
[junit4] 2> 2280783 T7761 oasc.ZkController.createCollectionZkNode Check
for collection zkNode:control_collection
[junit4] 2> 2280784 T7761 oasc.ZkController.createCollectionZkNode
Collection zkNode exists
[junit4] 2> 2280784 T7761 oascc.ZkStateReader.readConfigName Load
collection config from:/collections/control_collection
[junit4] 2> 2280785 T7761 oascc.ZkStateReader.readConfigName
path=/collections/control_collection configName=conf1 specified config exists
in ZooKeeper
[junit4] 2> 2280786 T7761 oasc.SolrResourceLoader.<init> new
SolrResourceLoader for directory:
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/collection1/'
[junit4] 2> 2280787 T7761 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/collection1/lib/.svn/'
to classloader
[junit4] 2> 2280788 T7761 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/collection1/lib/classes/'
to classloader
[junit4] 2> 2280788 T7761 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/collection1/lib/README'
to classloader
[junit4] 2> 2280840 T7761 oasc.SolrConfig.<init> Using Lucene
MatchVersion: 6.0.0
[junit4] 2> 2280863 T7761 oasc.SolrConfig.<init> Loaded SolrConfig:
solrconfig.xml
[junit4] 2> 2280865 T7761 oass.IndexSchema.readSchema Reading Solr Schema
from schema.xml
[junit4] 2> 2280899 T7761 oass.IndexSchema.readSchema [collection1] Schema
name=test
[junit4] 2> 2281104 T7761 oass.IndexSchema.readSchema default search field
in schema is text
[junit4] 2> 2281107 T7761 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2281109 T7761 oass.FileExchangeRateProvider.reload Reloading
exchange rates from file currency.xml
[junit4] 2> 2281114 T7761 oass.FileExchangeRateProvider.reload Reloading
exchange rates from file currency.xml
[junit4] 2> 2281161 T7761 oasc.CoreContainer.create Creating SolrCore
'collection1' using configuration from collection control_collection
[junit4] 2> 2281161 T7761 oasc.SolrCore.initDirectoryFactory
solr.StandardDirectoryFactory
[junit4] 2> 2281162 T7761 oasc.SolrCore.<init> [collection1] Opening new
SolrCore at
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-002/collection1/,
dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data/
[junit4] 2> 2281162 T7761 oasc.JmxMonitoredMap.<init> JMX monitoring is
enabled. Adding Solr mbeans to JMX Server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1e3fad
[junit4] 2> 2281164 T7761 oasc.CachingDirectoryFactory.get return new
directory for
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data
[junit4] 2> 2281165 T7761 oasc.SolrCore.getNewIndexDir New index directory
detected: old=null
new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data/index/
[junit4] 2> 2281165 T7761 oasc.SolrCore.initIndex WARN [collection1] Solr
index directory
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data/index'
doesn't exist. Creating new index...
[junit4] 2> 2281166 T7761 oasc.CachingDirectoryFactory.get return new
directory for
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data/index
[junit4] 2> 2281167 T7761 oasu.RandomMergePolicy.<init> RandomMergePolicy
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.3365435001901582]
[junit4] 2> 2281171 T7761 oasc.SolrDeletionPolicy.onCommit
SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2>
commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2281171 T7761 oasc.SolrDeletionPolicy.updateCommits newest
commit generation = 1
[junit4] 2> 2281177 T7761 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "nodistrib"
[junit4] 2> 2281178 T7761 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "dedupe"
[junit4] 2> 2281178 T7761 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2281179 T7761 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "stored_sig"
[junit4] 2> 2281179 T7761 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2281179 T7761 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2281180 T7761 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2281180 T7761 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain
"distrib-dup-test-chain-implicit"
[junit4] 2> 2281180 T7761 oasc.SolrCore.loadUpdateProcessorChains no
updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2281182 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2281182 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2281183 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2281184 T7761 oasc.RequestHandlers.initHandlersFromConfig
created standard: solr.StandardRequestHandler
[junit4] 2> 2281185 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /get: solr.RealTimeGetHandler
[junit4] 2> 2281185 T7761 oasc.RequestHandlers.initHandlersFromConfig
adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2281186 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /replication: solr.ReplicationHandler
[junit4] 2> 2281187 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /update: solr.UpdateRequestHandler
[junit4] 2> 2281187 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2281189 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2281191 T7761 oasc.RequestHandlers.initHandlersFromConfig
created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2281195 T7761 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2281197 T7761 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2281198 T7761 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2281200 T7761 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2281202 T7761 oasu.CommitTracker.<init> Hard AutoCommit:
disabled
[junit4] 2> 2281203 T7761 oasu.CommitTracker.<init> Soft AutoCommit:
disabled
[junit4] 2> 2281205 T7761 oasu.RandomMergePolicy.<init> RandomMergePolicy
wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=39, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=5.00390625,
floorSegmentMB=0.5380859375, forceMergeDeletesPctAllowed=26.453403245126665,
segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.8020713062704511
[junit4] 2> 2281206 T7761 oasc.SolrDeletionPolicy.onInit
SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2>
commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2281206 T7761 oasc.SolrDeletionPolicy.updateCommits newest
commit generation = 1
[junit4] 2> 2281207 T7761 oass.SolrIndexSearcher.<init> Opening
Searcher@e823e8[collection1] main
[junit4] 2> 2281207 T7761 oascc.ZkStateReader.readConfigName Load
collection config from:/collections/control_collection
[junit4] 2> 2281209 T7761 oascc.ZkStateReader.readConfigName
path=/collections/control_collection configName=conf1 specified config exists
in ZooKeeper
[junit4] 2> 2281209 T7761 oasr.ManagedResourceStorage.newStorageIO Setting
up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2281210 T7761
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2281210 T7761 oasr.RestManager.init Initializing RestManager
with initArgs: {}
[junit4] 2> 2281210 T7761 oasr.ManagedResourceStorage.load Reading
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2281211 T7761
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found
for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2281212 T7761 oasr.ManagedResourceStorage.load Loaded null at
path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2281212 T7761 oascc.SolrZkClient.makePath makePath:
/configs/conf1/_rest_managed.json
[junit4] 2> 2281219 T7761
oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new
znode /configs/conf1/_rest_managed.json
[junit4] 2> 2281219 T7761 oasr.ManagedResourceStorage$JsonStorage.store
Saved JSON object to path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2281219 T7761 oasr.RestManager.init Initializing 0 registered
ManagedResources
[junit4] 2> 2281230 T7763 oasc.SolrCore.registerSearcher [collection1]
Registered new searcher Searcher@e823e8[collection1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2281232 T7761 oasc.CoreContainer.registerCore registering
core: collection1
[junit4] 2> 2281233 T7766 oasc.ZkController.register Register replica -
core:collection1 address:http://127.0.0.1:49660 collection:control_collection
shard:shard1
[junit4] 2> 2281234 T7735 oass.SolrDispatchFilter.init
user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 2281234 T7735 oass.SolrDispatchFilter.init
SolrDispatchFilter.init() done
[junit4] 2> 2281235 T7735
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
ZkCredentialsProvider
[junit4] 2> 2281236 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2281236 T7766 oascc.SolrZkClient.makePath makePath:
/collections/control_collection/leader_elect/shard1/election
[junit4] 2> 2281237 T7768 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1c36625 name:ZooKeeperConnection
Watcher:127.0.0.1:52726/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2281238 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2281238 T7735 oascc.SolrZkClient.createZkACLProvider Using
default ZkACLProvider
[junit4] 2> 2281240 T7735
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2281244 T7766 oasc.ShardLeaderElectionContext.runLeaderProcess
Running the leader process for shard shard1
[junit4] 2> 2281247 T7735 oasc.ChaosMonkey.monkeyLog monkey: init - expire
sessions:false cause connection loss:false
[junit4] 2> 2281248 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2281249 T7766
oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found
to continue.
[junit4] 2> 2281251 T7766 oasc.ShardLeaderElectionContext.runLeaderProcess
I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3133 name=collection1
org.apache.solr.core.SolrCore@19a362c url=http://127.0.0.1:49660/collection1
node=127.0.0.1:49660_ C3133_STATE=coll:control_collection core:collection1
props:{core=collection1, base_url=http://127.0.0.1:49660,
node_name=127.0.0.1:49660_, state=down}
[junit4] 2> 2281254 T7766 C3133 P49660 oasc.SyncStrategy.sync Sync
replicas to http://127.0.0.1:49660/collection1/
[junit4] 2> 2281256 T7766 C3133 P49660 oasc.SyncStrategy.syncReplicas Sync
Success - now sync replicas to me
[junit4] 2> 2281256 T7766 C3133 P49660 oasc.SyncStrategy.syncToMe
http://127.0.0.1:49660/collection1/ has no replicas
[junit4] 2> 2281256 T7766 oasc.ShardLeaderElectionContext.runLeaderProcess
I am the new leader: http://127.0.0.1:49660/collection1/ shard1
[junit4] 2> 2281257 T7766 oascc.SolrZkClient.makePath makePath:
/collections/control_collection/leaders/shard1
[junit4] 2> 2281263 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2281368 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2281368 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2281415 T7766 oasc.ZkController.register We are
http://127.0.0.1:49660/collection1/ and leader is
http://127.0.0.1:49660/collection1/
[junit4] 2> 2281416 T7766 oasc.ZkController.register No LogReplay needed
for core=collection1 baseURL=http://127.0.0.1:49660
[junit4] 2> 2281416 T7766 oasc.ZkController.checkRecovery I am the leader,
no recovery necessary
[junit4] 2> 2281416 T7766 oasc.ZkController.publish publishing
core=collection1 state=active collection=control_collection
[junit4] 2> 2281417 T7766 oasc.ZkController.publish numShards not found on
descriptor - reading it from system property
[junit4] 2> 2281418 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2281424 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:49660";,
[junit4] 2> "node_name":"127.0.0.1:49660_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "operation":"state"}
[junit4] 2> 2281528 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2281529 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2281595 T7735 oasc.AbstractFullDistribZkTestBase.createJettys
create jetty 1
[junit4] 2> 2281596 T7735 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2281598 T7735 oejs.AbstractConnector.doStart Started
[email protected]:54536
[junit4] 2> 2281602 T7735 oass.SolrDispatchFilter.init
SolrDispatchFilter.init()
[junit4] 2> 2281603 T7735 oasc.SolrResourceLoader.locateSolrHome JNDI not
configured for solr (NoInitialContextEx)
[junit4] 2> 2281603 T7735 oasc.SolrResourceLoader.locateSolrHome using
system property solr.solr.home:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003
[junit4] 2> 2281604 T7735 oasc.SolrResourceLoader.<init> new
SolrResourceLoader for directory:
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/'
[junit4] 2> 2281629 T7735 oasc.ConfigSolr.fromFile Loading container
configuration from
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/solr.xml
[junit4] 2> 2281643 T7735 oasc.CoreContainer.<init> New CoreContainer
27955044
[junit4] 2> 2281644 T7735 oasc.CoreContainer.load Loading cores into
CoreContainer
[instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/]
[junit4] 2> 2281644 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting socketTimeout to: 90000
[junit4] 2> 2281645 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting urlScheme to:
[junit4] 2> 2281645 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting connTimeout to: 15000
[junit4] 2> 2281645 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnectionsPerHost to: 20
[junit4] 2> 2281646 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnections to: 10000
[junit4] 2> 2281646 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting corePoolSize to: 0
[junit4] 2> 2281646 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maximumPoolSize to: 2147483647
[junit4] 2> 2281647 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxThreadIdleTime to: 5
[junit4] 2> 2281647 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting sizeOfQueue to: -1
[junit4] 2> 2281647 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting fairnessPolicy to: false
[junit4] 2> 2281648 T7735 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params:
socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2281648 T7735 oasl.LogWatcher.createWatcher SLF4J impl is
org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2281649 T7735 oasl.LogWatcher.newRegisteredLogWatcher
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2281649 T7735 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2281650 T7735 oasc.ZkContainer.initZooKeeper Zookeeper
client=127.0.0.1:52726/solr
[junit4] 2> 2281650 T7735 oasc.ZkController.checkChrootPath zkHost
includes chroot
[junit4] 2> 2281650 T7735
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
ZkCredentialsProvider
[junit4] 2> 2281651 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2281652 T7780 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@d715ec name:ZooKeeperConnection
Watcher:127.0.0.1:52726 got event WatchedEvent state:SyncConnected type:None
path:null path:null type:None
[junit4] 2> 2281653 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2281653 T7735 oascc.SolrZkClient.createZkACLProvider Using
default ZkACLProvider
[junit4] 2> 2281655 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2281656 T7782 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@131919b name:ZooKeeperConnection
Watcher:127.0.0.1:52726/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2281656 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2281662 T7735
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2282666 T7735 oasc.ZkController.createEphemeralLiveNode
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54536_
[junit4] 2> 2282668 T7735 oascc.SolrZkClient.makePath makePath:
/live_nodes/127.0.0.1:54536_
[junit4] 2> 2282674 T7735 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2282679 T7784 oasc.ZkController.publish publishing
core=collection1 state=down collection=collection1
[junit4] 2> 2282680 T7784 oasc.ZkController.publish numShards not found on
descriptor - reading it from system property
[junit4] 2> 2282681 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2282681 T7784 oasc.ZkController.waitForCoreNodeName look for
our core node name
[junit4] 2> 2282685 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:54536";,
[junit4] 2> "node_name":"127.0.0.1:54536_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 2282685 T7758
oasc.Overseer$ClusterStateUpdater.createCollection Create collection
collection1 with shards [shard1]
[junit4] 2> 2282685 T7758
oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
[junit4] 2> 2282686 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard1
[junit4] 2> 2282790 T7783 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2282790 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2282790 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2283682 T7784 oasc.ZkController.waitForShardId waiting to find
shard id in clusterstate for collection1
[junit4] 2> 2283682 T7784 oasc.ZkController.createCollectionZkNode Check
for collection zkNode:collection1
[junit4] 2> 2283683 T7784 oasc.ZkController.createCollectionZkNode
Collection zkNode exists
[junit4] 2> 2283683 T7784 oascc.ZkStateReader.readConfigName Load
collection config from:/collections/collection1
[junit4] 2> 2283685 T7784 oascc.ZkStateReader.readConfigName
path=/collections/collection1 configName=conf1 specified config exists in
ZooKeeper
[junit4] 2> 2283685 T7784 oasc.SolrResourceLoader.<init> new
SolrResourceLoader for directory:
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/collection1/'
[junit4] 2> 2283686 T7784 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/collection1/lib/.svn/'
to classloader
[junit4] 2> 2283687 T7784 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/collection1/lib/classes/'
to classloader
[junit4] 2> 2283687 T7784 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/collection1/lib/README'
to classloader
[junit4] 2> 2283727 T7784 oasc.SolrConfig.<init> Using Lucene
MatchVersion: 6.0.0
[junit4] 2> 2283742 T7784 oasc.SolrConfig.<init> Loaded SolrConfig:
solrconfig.xml
[junit4] 2> 2283744 T7784 oass.IndexSchema.readSchema Reading Solr Schema
from schema.xml
[junit4] 2> 2283755 T7784 oass.IndexSchema.readSchema [collection1] Schema
name=test
[junit4] 2> 2283905 T7784 oass.IndexSchema.readSchema default search field
in schema is text
[junit4] 2> 2283907 T7784 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2283908 T7784 oass.FileExchangeRateProvider.reload Reloading
exchange rates from file currency.xml
[junit4] 2> 2283912 T7784 oass.FileExchangeRateProvider.reload Reloading
exchange rates from file currency.xml
[junit4] 2> 2283941 T7784 oasc.CoreContainer.create Creating SolrCore
'collection1' using configuration from collection collection1
[junit4] 2> 2283942 T7784 oasc.SolrCore.initDirectoryFactory
solr.StandardDirectoryFactory
[junit4] 2> 2283942 T7784 oasc.SolrCore.<init> [collection1] Opening new
SolrCore at
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-003/collection1/,
dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1/
[junit4] 2> 2283942 T7784 oasc.JmxMonitoredMap.<init> JMX monitoring is
enabled. Adding Solr mbeans to JMX Server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1e3fad
[junit4] 2> 2283944 T7784 oasc.CachingDirectoryFactory.get return new
directory for
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1
[junit4] 2> 2283945 T7784 oasc.SolrCore.getNewIndexDir New index directory
detected: old=null
new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1/index/
[junit4] 2> 2283945 T7784 oasc.SolrCore.initIndex WARN [collection1] Solr
index directory
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1/index'
doesn't exist. Creating new index...
[junit4] 2> 2283946 T7784 oasc.CachingDirectoryFactory.get return new
directory for
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1/index
[junit4] 2> 2283946 T7784 oasu.RandomMergePolicy.<init> RandomMergePolicy
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.3365435001901582]
[junit4] 2> 2283955 T7784 oasc.SolrDeletionPolicy.onCommit
SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2>
commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 2283956 T7784 oasc.SolrDeletionPolicy.updateCommits newest
commit generation = 1
[junit4] 2> 2283964 T7784 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "nodistrib"
[junit4] 2> 2283965 T7784 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "dedupe"
[junit4] 2> 2283965 T7784 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2283966 T7784 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "stored_sig"
[junit4] 2> 2283966 T7784 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2283967 T7784 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2283967 T7784 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2283968 T7784 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain
"distrib-dup-test-chain-implicit"
[junit4] 2> 2283968 T7784 oasc.SolrCore.loadUpdateProcessorChains no
updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2283969 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2283971 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2283972 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2283973 T7784 oasc.RequestHandlers.initHandlersFromConfig
created standard: solr.StandardRequestHandler
[junit4] 2> 2283974 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /get: solr.RealTimeGetHandler
[junit4] 2> 2283974 T7784 oasc.RequestHandlers.initHandlersFromConfig
adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2283975 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /replication: solr.ReplicationHandler
[junit4] 2> 2283976 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /update: solr.UpdateRequestHandler
[junit4] 2> 2283976 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2283977 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2283978 T7784 oasc.RequestHandlers.initHandlersFromConfig
created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2283980 T7784 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2283982 T7784 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2283983 T7784 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2283984 T7784 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2283985 T7784 oasu.CommitTracker.<init> Hard AutoCommit:
disabled
[junit4] 2> 2283986 T7784 oasu.CommitTracker.<init> Soft AutoCommit:
disabled
[junit4] 2> 2283987 T7784 oasu.RandomMergePolicy.<init> RandomMergePolicy
wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=39, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=5.00390625,
floorSegmentMB=0.5380859375, forceMergeDeletesPctAllowed=26.453403245126665,
segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.8020713062704511
[junit4] 2> 2283988 T7784 oasc.SolrDeletionPolicy.onInit
SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2>
commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 2283988 T7784 oasc.SolrDeletionPolicy.updateCommits newest
commit generation = 1
[junit4] 2> 2283989 T7784 oass.SolrIndexSearcher.<init> Opening
Searcher@1f28a[collection1] main
[junit4] 2> 2283989 T7784 oascc.ZkStateReader.readConfigName Load
collection config from:/collections/collection1
[junit4] 2> 2283991 T7784 oascc.ZkStateReader.readConfigName
path=/collections/collection1 configName=conf1 specified config exists in
ZooKeeper
[junit4] 2> 2283991 T7784 oasr.ManagedResourceStorage.newStorageIO Setting
up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2283992 T7784
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2283992 T7784 oasr.RestManager.init Initializing RestManager
with initArgs: {}
[junit4] 2> 2283992 T7784 oasr.ManagedResourceStorage.load Reading
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2283994 T7784
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes
from znode /configs/conf1/_rest_managed.json
[junit4] 2> 2283994 T7784 oasr.ManagedResourceStorage.load Loaded
LinkedHashMap at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2283994 T7784 oasr.ManagedResource.reloadFromStorage Loaded
initArgs {} for /rest/managed
[junit4] 2> 2283995 T7784 oasr.RestManager.init Initializing 0 registered
ManagedResources
[junit4] 2> 2284001 T7785 oasc.SolrCore.registerSearcher [collection1]
Registered new searcher Searcher@1f28a[collection1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2284009 T7784 oasc.CoreContainer.registerCore registering
core: collection1
[junit4] 2> 2284010 T7788 oasc.ZkController.register Register replica -
core:collection1 address:http://127.0.0.1:54536 collection:collection1
shard:shard1
[junit4] 2> 2284010 T7735 oass.SolrDispatchFilter.init
user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 2284011 T7788 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2284011 T7735 oass.SolrDispatchFilter.init
SolrDispatchFilter.init() done
[junit4] 2> 2284016 T7788 oasc.ShardLeaderElectionContext.runLeaderProcess
Running the leader process for shard shard1
[junit4] 2> 2284019 T7788
oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found
to continue.
[junit4] 2> 2284019 T7788 oasc.ShardLeaderElectionContext.runLeaderProcess
I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3134 name=collection1
org.apache.solr.core.SolrCore@3bda1b url=http://127.0.0.1:54536/collection1
node=127.0.0.1:54536_ C3134_STATE=coll:collection1 core:collection1
props:{core=collection1, base_url=http://127.0.0.1:54536,
node_name=127.0.0.1:54536_, state=down}
[junit4] 2> 2284019 T7788 C3134 P54536 oasc.SyncStrategy.sync Sync
replicas to http://127.0.0.1:54536/collection1/
[junit4] 2> 2284019 T7788 C3134 P54536 oasc.SyncStrategy.syncReplicas Sync
Success - now sync replicas to me
[junit4] 2> 2284020 T7788 C3134 P54536 oasc.SyncStrategy.syncToMe
http://127.0.0.1:54536/collection1/ has no replicas
[junit4] 2> 2284020 T7788 oasc.ShardLeaderElectionContext.runLeaderProcess
I am the new leader: http://127.0.0.1:54536/collection1/ shard1
[junit4] 2> 2284020 T7788 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2284023 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2284034 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2284035 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2284035 T7783 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2284035 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2284077 T7788 oasc.ZkController.register We are
http://127.0.0.1:54536/collection1/ and leader is
http://127.0.0.1:54536/collection1/
[junit4] 2> 2284078 T7788 oasc.ZkController.register No LogReplay needed
for core=collection1 baseURL=http://127.0.0.1:54536
[junit4] 2> 2284078 T7788 oasc.ZkController.checkRecovery I am the leader,
no recovery necessary
[junit4] 2> 2284078 T7788 oasc.ZkController.publish publishing
core=collection1 state=active collection=collection1
[junit4] 2> 2284078 T7788 oasc.ZkController.publish numShards not found on
descriptor - reading it from system property
[junit4] 2> 2284081 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2284086 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:54536";,
[junit4] 2> "node_name":"127.0.0.1:54536_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 2284190 T7783 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2284190 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2284190 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2284327 T7735 oasc.AbstractFullDistribZkTestBase.createJettys
create jetty 2
[junit4] 2> 2284328 T7735 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2284330 T7735 oejs.AbstractConnector.doStart Started
[email protected]:42702
[junit4] 2> 2284332 T7735 oass.SolrDispatchFilter.init
SolrDispatchFilter.init()
[junit4] 2> 2284333 T7735 oasc.SolrResourceLoader.locateSolrHome JNDI not
configured for solr (NoInitialContextEx)
[junit4] 2> 2284334 T7735 oasc.SolrResourceLoader.locateSolrHome using
system property solr.solr.home:
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004
[junit4] 2> 2284335 T7735 oasc.SolrResourceLoader.<init> new
SolrResourceLoader for directory:
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/'
[junit4] 2> 2284365 T7735 oasc.ConfigSolr.fromFile Loading container
configuration from
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/solr.xml
[junit4] 2> 2284379 T7735 oasc.CoreContainer.<init> New CoreContainer
18947403
[junit4] 2> 2284380 T7735 oasc.CoreContainer.load Loading cores into
CoreContainer
[instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/]
[junit4] 2> 2284380 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting socketTimeout to: 90000
[junit4] 2> 2284381 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting urlScheme to:
[junit4] 2> 2284381 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting connTimeout to: 15000
[junit4] 2> 2284381 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnectionsPerHost to: 20
[junit4] 2> 2284382 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxConnections to: 10000
[junit4] 2> 2284382 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting corePoolSize to: 0
[junit4] 2> 2284383 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maximumPoolSize to: 2147483647
[junit4] 2> 2284383 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting maxThreadIdleTime to: 5
[junit4] 2> 2284383 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting sizeOfQueue to: -1
[junit4] 2> 2284384 T7735 oashc.HttpShardHandlerFactory.getParameter
Setting fairnessPolicy to: false
[junit4] 2> 2284384 T7735 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params:
socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2284385 T7735 oasl.LogWatcher.createWatcher SLF4J impl is
org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2284385 T7735 oasl.LogWatcher.newRegisteredLogWatcher
Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2284385 T7735 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2284386 T7735 oasc.ZkContainer.initZooKeeper Zookeeper
client=127.0.0.1:52726/solr
[junit4] 2> 2284386 T7735 oasc.ZkController.checkChrootPath zkHost
includes chroot
[junit4] 2> 2284387 T7735
oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
ZkCredentialsProvider
[junit4] 2> 2284387 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2284389 T7799 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@3d20f7 name:ZooKeeperConnection
Watcher:127.0.0.1:52726 got event WatchedEvent state:SyncConnected type:None
path:null path:null type:None
[junit4] 2> 2284389 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2284390 T7735 oascc.SolrZkClient.createZkACLProvider Using
default ZkACLProvider
[junit4] 2> 2284392 T7735 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2284393 T7801 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@8ef71e name:ZooKeeperConnection
Watcher:127.0.0.1:52726/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2284394 T7735 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2284399 T7735
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2285410 T7735 oasc.ZkController.createEphemeralLiveNode
Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42702_
[junit4] 2> 2285411 T7735 oascc.SolrZkClient.makePath makePath:
/live_nodes/127.0.0.1:42702_
[junit4] 2> 2285415 T7735 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2285421 T7803 oasc.ZkController.publish publishing
core=collection1 state=down collection=collection1
[junit4] 2> 2285421 T7803 oasc.ZkController.publish numShards not found on
descriptor - reading it from system property
[junit4] 2> 2285423 T7803 oasc.ZkController.waitForCoreNodeName look for
our core node name
[junit4] 2> 2285423 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2285427 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42702";,
[junit4] 2> "node_name":"127.0.0.1:42702_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"down",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 2285428 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Collection already exists with numShards=1
[junit4] 2> 2285428 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard1
[junit4] 2> 2285533 T7783 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2285533 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2285533 T7802 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2285533 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2286423 T7803 oasc.ZkController.waitForShardId waiting to find
shard id in clusterstate for collection1
[junit4] 2> 2286424 T7803 oasc.ZkController.createCollectionZkNode Check
for collection zkNode:collection1
[junit4] 2> 2286425 T7803 oasc.ZkController.createCollectionZkNode
Collection zkNode exists
[junit4] 2> 2286425 T7803 oascc.ZkStateReader.readConfigName Load
collection config from:/collections/collection1
[junit4] 2> 2286429 T7803 oascc.ZkStateReader.readConfigName
path=/collections/collection1 configName=conf1 specified config exists in
ZooKeeper
[junit4] 2> 2286429 T7803 oasc.SolrResourceLoader.<init> new
SolrResourceLoader for directory:
'../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/collection1/'
[junit4] 2> 2286430 T7803 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/collection1/lib/.svn/'
to classloader
[junit4] 2> 2286431 T7803 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/collection1/lib/classes/'
to classloader
[junit4] 2> 2286431 T7803 oasc.SolrResourceLoader.replaceClassLoader
Adding
'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/collection1/lib/README'
to classloader
[junit4] 2> 2286469 T7803 oasc.SolrConfig.<init> Using Lucene
MatchVersion: 6.0.0
[junit4] 2> 2286484 T7803 oasc.SolrConfig.<init> Loaded SolrConfig:
solrconfig.xml
[junit4] 2> 2286485 T7803 oass.IndexSchema.readSchema Reading Solr Schema
from schema.xml
[junit4] 2> 2286497 T7803 oass.IndexSchema.readSchema [collection1] Schema
name=test
[junit4] 2> 2286613 T7803 oass.IndexSchema.readSchema default search field
in schema is text
[junit4] 2> 2286615 T7803 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2286616 T7803 oass.FileExchangeRateProvider.reload Reloading
exchange rates from file currency.xml
[junit4] 2> 2286619 T7803 oass.FileExchangeRateProvider.reload Reloading
exchange rates from file currency.xml
[junit4] 2> 2286646 T7803 oasc.CoreContainer.create Creating SolrCore
'collection1' using configuration from collection collection1
[junit4] 2> 2286646 T7803 oasc.SolrCore.initDirectoryFactory
solr.StandardDirectoryFactory
[junit4] 2> 2286647 T7803 oasc.SolrCore.<init> [collection1] Opening new
SolrCore at
../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-004/collection1/,
dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/
[junit4] 2> 2286647 T7803 oasc.JmxMonitoredMap.<init> JMX monitoring is
enabled. Adding Solr mbeans to JMX Server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1e3fad
[junit4] 2> 2286648 T7803 oasc.CachingDirectoryFactory.get return new
directory for
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2
[junit4] 2> 2286649 T7803 oasc.SolrCore.getNewIndexDir New index directory
detected: old=null
new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index/
[junit4] 2> 2286649 T7803 oasc.SolrCore.initIndex WARN [collection1] Solr
index directory
'/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index'
doesn't exist. Creating new index...
[junit4] 2> 2286649 T7803 oasc.CachingDirectoryFactory.get return new
directory for
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index
[junit4] 2> 2286650 T7803 oasu.RandomMergePolicy.<init> RandomMergePolicy
wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy:
minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.3365435001901582]
[junit4] 2> 2286656 T7803 oasc.SolrDeletionPolicy.onCommit
SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2>
commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 2286657 T7803 oasc.SolrDeletionPolicy.updateCommits newest
commit generation = 1
[junit4] 2> 2286666 T7803 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "nodistrib"
[junit4] 2> 2286667 T7803 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "dedupe"
[junit4] 2> 2286667 T7803 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2286667 T7803 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "stored_sig"
[junit4] 2> 2286668 T7803 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2286668 T7803 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2286669 T7803 oasup.UpdateRequestProcessorChain.init creating
updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2286669 T7803 oasup.UpdateRequestProcessorChain.init inserting
DistributedUpdateProcessorFactory into updateRequestProcessorChain
"distrib-dup-test-chain-implicit"
[junit4] 2> 2286669 T7803 oasc.SolrCore.loadUpdateProcessorChains no
updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2286670 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2286671 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2286672 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2286673 T7803 oasc.RequestHandlers.initHandlersFromConfig
created standard: solr.StandardRequestHandler
[junit4] 2> 2286673 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /get: solr.RealTimeGetHandler
[junit4] 2> 2286674 T7803 oasc.RequestHandlers.initHandlersFromConfig
adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2286674 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /replication: solr.ReplicationHandler
[junit4] 2> 2286675 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /update: solr.UpdateRequestHandler
[junit4] 2> 2286675 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2286676 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2286677 T7803 oasc.RequestHandlers.initHandlersFromConfig
created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2286679 T7803 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2286680 T7803 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2286681 T7803 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2286682 T7803 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2286684 T7803 oasu.CommitTracker.<init> Hard AutoCommit:
disabled
[junit4] 2> 2286684 T7803 oasu.CommitTracker.<init> Soft AutoCommit:
disabled
[junit4] 2> 2286686 T7803 oasu.RandomMergePolicy.<init> RandomMergePolicy
wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy:
maxMergeAtOnce=39, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=5.00390625,
floorSegmentMB=0.5380859375, forceMergeDeletesPctAllowed=26.453403245126665,
segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.8020713062704511
[junit4] 2> 2286686 T7803 oasc.SolrDeletionPolicy.onInit
SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2>
commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 2286687 T7803 oasc.SolrDeletionPolicy.updateCommits newest
commit generation = 1
[junit4] 2> 2286687 T7803 oass.SolrIndexSearcher.<init> Opening
Searcher@9d3f7a[collection1] main
[junit4] 2> 2286687 T7803 oascc.ZkStateReader.readConfigName Load
collection config from:/collections/collection1
[junit4] 2> 2286688 T7803 oascc.ZkStateReader.readConfigName
path=/collections/collection1 configName=conf1 specified config exists in
ZooKeeper
[junit4] 2> 2286688 T7803 oasr.ManagedResourceStorage.newStorageIO Setting
up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2286689 T7803
oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2286689 T7803 oasr.RestManager.init Initializing RestManager
with initArgs: {}
[junit4] 2> 2286690 T7803 oasr.ManagedResourceStorage.load Reading
_rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2286691 T7803
oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes
from znode /configs/conf1/_rest_managed.json
[junit4] 2> 2286691 T7803 oasr.ManagedResourceStorage.load Loaded
LinkedHashMap at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2286691 T7803 oasr.ManagedResource.reloadFromStorage Loaded
initArgs {} for /rest/managed
[junit4] 2> 2286692 T7803 oasr.RestManager.init Initializing 0 registered
ManagedResources
[junit4] 2> 2286700 T7804 oasc.SolrCore.registerSearcher [collection1]
Registered new searcher Searcher@9d3f7a[collection1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2286708 T7803 oasc.CoreContainer.registerCore registering
core: collection1
[junit4] 2> 2286709 T7807 oasc.ZkController.register Register replica -
core:collection1 address:http://127.0.0.1:42702 collection:collection1
shard:shard1
[junit4] 2> 2286710 T7735 oass.SolrDispatchFilter.init
user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 2286710 T7735 oass.SolrDispatchFilter.init
SolrDispatchFilter.init() done
[junit4] 2> 2286713 T7807 oasc.ZkController.register We are
http://127.0.0.1:42702/collection1/ and leader is
http://127.0.0.1:54536/collection1/
[junit4] 2> 2286713 T7807 oasc.ZkController.register No LogReplay needed
for core=collection1 baseURL=http://127.0.0.1:42702
[junit4] 2> 2286713 T7735
oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to
finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4] 2> 2286713 T7807 oasc.ZkController.checkRecovery Core needs to
recover:collection1
[junit4] 2> 2286714 T7807 oasu.DefaultSolrCoreState.doRecovery Running
recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C3135 name=collection1
org.apache.solr.core.SolrCore@524c28 url=http://127.0.0.1:42702/collection1
node=127.0.0.1:42702_ C3135_STATE=coll:collection1 core:collection1
props:{core=collection1, base_url=http://127.0.0.1:42702,
node_name=127.0.0.1:42702_, state=down}
[junit4] 2> 2286715 T7808 C3135 P42702 oasc.RecoveryStrategy.run Starting
recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 2286715 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
###### startupVersions=[]
[junit4] 2> 2286716 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
Publishing state of core collection1 as recovering, leader is
http://127.0.0.1:54536/collection1/ and I am http://127.0.0.1:42702/collection1/
[junit4] 2> 2286716 T7808 C3135 P42702 oasc.ZkController.publish
publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 2286716 T7808 C3135 P42702 oasc.ZkController.publish numShards
not found on descriptor - reading it from system property
[junit4] 2> 2286717 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2286718 T7808 C3135 P42702
oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to
http://127.0.0.1:54536; WaitForState:
action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A42702_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
[junit4] 2> 2286720 T7777 oasha.CoreAdminHandler.handleWaitForStateAction
Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true,
onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 2286721 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42702";,
[junit4] 2> "node_name":"127.0.0.1:42702_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"recovering",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 2286723 T7777 oasha.CoreAdminHandler.handleWaitForStateAction
Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have
state: recovering
[junit4] 2> 2286724 T7777 oasha.CoreAdminHandler.handleWaitForStateAction
In WaitForState(recovering): collection=collection1, shard=shard1,
thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true,
live=true, checkLive=true, currentState=down, localState=active,
nodeName=127.0.0.1:42702_, coreNodeName=core_node2,
onlyIfActiveCheckResult=false, nodeProps:
core_node2:{"core":"collection1","base_url":"http://127.0.0.1:42702","node_name":"127.0.0.1:42702_","state":"down"}
[junit4] 2> 2286825 T7783 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2286825 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2286825 T7802 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2286825 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2287724 T7777 oasha.CoreAdminHandler.handleWaitForStateAction
In WaitForState(recovering): collection=collection1, shard=shard1,
thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true,
live=true, checkLive=true, currentState=recovering, localState=active,
nodeName=127.0.0.1:42702_, coreNodeName=core_node2,
onlyIfActiveCheckResult=false, nodeProps:
core_node2:{"core":"collection1","base_url":"http://127.0.0.1:42702","node_name":"127.0.0.1:42702_","state":"recovering"}
[junit4] 2> 2287725 T7777 oasha.CoreAdminHandler.handleWaitForStateAction
Waited coreNodeName: core_node2, state: recovering, checkLive: true,
onlyIfLeader: true for: 1 seconds.
[junit4] 2> 2287725 T7777 oass.SolrDispatchFilter.handleAdminRequest
[admin] webapp=null path=/admin/cores
params={nodeName=127.0.0.1:42702_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
status=0 QTime=1005
[junit4] 2> C3135_STATE=coll:collection1 core:collection1
props:{core=collection1, base_url=http://127.0.0.1:42702,
node_name=127.0.0.1:42702_, state=recovering}
[junit4] 2> 2289727 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
Attempting to PeerSync from http://127.0.0.1:54536/collection1/
core=collection1 - recoveringAfterStartup=true
[junit4] 2> 2289728 T7808 C3135 P42702 oasu.PeerSync.sync PeerSync:
core=collection1 url=http://127.0.0.1:42702 START
replicas=[http://127.0.0.1:54536/collection1/] nUpdates=100
[junit4] 2> 2289728 T7808 C3135 P42702 oasu.PeerSync.sync WARN no frame of
reference to tell if we've missed updates
[junit4] 2> 2289729 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
PeerSync Recovery was not successful - trying replication. core=collection1
[junit4] 2> 2289730 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
Starting Replication Recovery. core=collection1
[junit4] 2> 2289730 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
Begin buffering updates. core=collection1
[junit4] 2> 2289730 T7808 C3135 P42702 oasu.UpdateLog.bufferUpdates
Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 2289730 T7808 C3135 P42702 oasc.RecoveryStrategy.replicate
Attempting to replicate from http://127.0.0.1:54536/collection1/.
core=collection1
[junit4] 2> C3134_STATE=coll:collection1 core:collection1
props:{core=collection1, base_url=http://127.0.0.1:54536,
node_name=127.0.0.1:54536_, state=active, leader=true}
[junit4] 2> 2289731 T7776 C3134 P54536 oasc.SolrCore.execute [collection1]
webapp= path=/get
params={distrib=false&qt=/get&getVersions=100&wt=javabin&version=2} status=0
QTime=0
[junit4] 2> 2289733 T7775 C3134 P54536 oasu.DirectUpdateHandler2.commit
start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2289733 T7775 C3134 P54536 oasu.DirectUpdateHandler2.commit No
uncommitted changes. Skipping IW.commit.
[junit4] 2> 2289735 T7775 C3134 P54536 oasu.DirectUpdateHandler2.commit
end_commit_flush
[junit4] 2> 2289735 T7775 C3134 P54536 oasup.LogUpdateProcessor.finish
[collection1] webapp= path=/update
params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2}
{commit=} 0 2
[junit4] 2> 2289737 T7808 C3135 P42702 oash.ReplicationHandler.inform
Commits will be reserved for 10000
[junit4] 2> 2289738 T7808 C3135 P42702 oash.SnapPuller.<init> No value
set for 'pollInterval'. Timer Task not started.
[junit4] 2> 2289742 T7776 C3134 P54536 oash.ReplicationHandler.inform
Commits will be reserved for 10000
[junit4] 2> 2289743 T7776 C3134 P54536 oasc.SolrCore.execute [collection1]
webapp= path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=2
[junit4] 2> 2289745 T7808 C3135 P42702 oasc.RecoveryStrategy.replay No
replay needed. core=collection1
[junit4] 2> 2289745 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
Replication Recovery was successful - registering as Active. core=collection1
[junit4] 2> 2289745 T7808 C3135 P42702 oasc.ZkController.publish
publishing core=collection1 state=active collection=collection1
[junit4] 2> 2289746 T7808 C3135 P42702 oasc.ZkController.publish numShards
not found on descriptor - reading it from system property
[junit4] 2> 2289748 T7762 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2289748 T7808 C3135 P42702 oasc.RecoveryStrategy.doRecovery
Finished recovery process. core=collection1
[junit4] 2> 2289751 T7758 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "roles":null,
[junit4] 2> "base_url":"http://127.0.0.1:42702";,
[junit4] 2> "node_name":"127.0.0.1:42702_",
[junit4] 2> "numShards":"1",
[junit4] 2> "state":"active",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "operation":"state"}
[junit4] 2> 2289754 T7783 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2289754 T7769 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2289754 T7802 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2289755 T7762 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2290727 T7735
oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished -
collection: collection1
[junit4] 2> ASYNC NEW_CORE C3136 name=collection1
org.apache.solr.core.SolrCore@19a362c url=http://127.0.0.1:49660/collection1
node=127.0.0.1:49660_ C3136_STATE=coll:control_collection core:collection1
props:{core=collection1, base_url=http://127.0.0.1:49660,
node_name=127.0.0.1:49660_, state=active, leader=true}
[junit4] 2> 2290732 T7750 C3136 P49660 oasup.LogUpdateProcessor.finish
[collection1] webapp= path=/update params={wt=javabin&version=2}
{deleteByQuery=*:* (-1481526123256872960)} 0 3
[junit4] 2> C3135_STATE=coll:collection1 core:collection1
props:{core=collection1, base_url=http://127.0.0.1:42702,
node_name=127.0.0.1:42702_, state=active}
[junit4] 2> 2290741 T7794 C3135 P42702 oasup.LogUpdateProcessor.finish
[collection1] webapp= path=/update
params={update.distrib=FROMLEADER&_version_=-1481526123262115840&distrib.from=http://127.0.0.1:54536/collection1/&wt=javabin&version=2}
{deleteByQuery=*:* (-1481526123262115840)} 0 2
[junit4] 2> 2290742 T7776 C3134 P54536 oasup.LogUpdateProcessor.finish
[collection1] webapp= path=/update params={wt=javabin&version=2}
{deleteByQuery=*:* (-1481526123262115840)} 0 7
[junit4] 2> 2290745 T7735 oasc.ChaosMonkey.monkeyLog monkey: starting
[junit4] 2> 2290750 T7795 C3135 P42702 oasc.SolrCore.execute [collection1]
webapp= path=/select params={q=to+come&wt=javabin&version=2} hits=
[...truncated too long message...]
r-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index;done=false>>]
[junit4] 2> 2313595 T7735 oasc.CachingDirectoryFactory.close Closing
directory:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001/tempDir-001/jetty2/index
[junit4] 2> 2313596 T7735 oasc.Overseer.close Overseer
(id=92595381938290696-127.0.0.1:42702_-n_0000000002) closing
[junit4] 2> 2313598 T7801 oaz.ClientCnxn$EventThread.processEvent ERROR
Error while calling watcher java.util.concurrent.RejectedExecutionException:
Task java.util.concurrent.FutureTask@10b787e rejected from
java.util.concurrent.ThreadPoolExecutor@1170d65[Shutting down, pool size = 2,
active threads = 0, queued tasks = 0, completed tasks = 15]
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
[junit4] 2> at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:263)
[junit4] 2> at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 2313599 T7822 oasc.Overseer$ClusterStateUpdater.run Overseer
Loop exiting : 127.0.0.1:42702_
[junit4] 2> 2313622 T7735 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 2318611 T7735 oas.SolrTestCaseJ4.tearDown ###Ending
testDistribSearch
[junit4] 2> 2318613 T7735 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:52726 52726
[junit4] 2> 2319157 T7736 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:52726 52726
[junit4] 2> 2319159 T7736 oasc.ZkTestServer$ZKServerMain.runFromConfig
WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 2
/solr/overseer_elect/election/92595381938290691-127.0.0.1:49660_-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch
-Dtests.seed=46AACD3FB0232159 -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=lv -Dtests.timezone=Africa/Ndjamena -Dtests.file.encoding=UTF-8
[junit4] FAILURE 40.1s J0 | ChaosMonkeyNothingIsSafeTest.testDistribSearch
<<<
[junit4] > Throwable #1: java.lang.AssertionError: There were too many
update fails - we expect it can happen, but shouldn't easily
[junit4] > at
__randomizedtesting.SeedInfo.seed([46AACD3FB0232159:C74C4327C77C4165]:0)
[junit4] > at
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:223)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2319172 T7735 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-46AACD3FB0232159-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50):
{rnd_b=OrdsLucene41(blocksize=128), _version_=FST41, a_t=FST41,
a_i=OrdsLucene41(blocksize=128), id=OrdsLucene41(blocksize=128)}, docValues:{},
sim=DefaultSimilarity, locale=lv, timezone=Africa/Ndjamena
[junit4] 2> NOTE: Linux 3.13.0-36-generic i386/Oracle Corporation
1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=245930488,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestCollationFieldDocValues,
TestDefaultSearchFieldResource, SSLMigrationTest, XmlUpdateRequestHandlerTest,
TestArbitraryIndexDir, TestMaxScoreQueryParser, TestSuggestSpellingConverter,
StandardRequestHandlerTest, TestUniqueKeyFieldResource, TestSystemIdResolver,
TestCopyFieldCollectionResource, DistributedQueryComponentOptimizationTest,
TestFileDictionaryLookup, TestSolr4Spatial,
CollectionsAPIAsyncDistributedZkTest, TestLuceneMatchVersion,
TestWordDelimiterFilterFactory, TestInitQParser, TestFastWriter,
SystemInfoHandlerTest, TestSortingResponseWriter, AssignTest,
TestCoreDiscovery, TestStressVersions, HighlighterMaxOffsetTest,
NotRequiredUniqueKeyTest, TestStressUserVersions, JSONWriterTest,
TestJmxIntegration, PrimitiveFieldTypeTest, TestHashPartitioner,
TestSolrDeletionPolicy1, TestSearchPerf, ExitableDirectoryReaderTest,
FieldMutatingUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest,
TestAnalyzedSuggestions, TestLRUCache, TestLeaderElectionZkExpiry,
SyncSliceTest, TestSolrIndexConfig, TestAnalyzeInfixSuggestions,
DocumentBuilderTest, SignatureUpdateProcessorFactoryTest, TestStandardQParsers,
HdfsWriteToMultipleCollectionsTest, ShardRoutingTest, TermsComponentTest,
CustomCollectionTest, AtomicUpdatesTest, RecoveryZkTest,
ParsingFieldUpdateProcessorsTest, TestManagedSynonymFilterFactory,
TestRequestStatusCollectionAPI, TestChildDocTransformer, SuggesterWFSTTest,
TestManagedSchemaDynamicFieldResource, ZkCLITest, HighlighterConfigTest,
StressHdfsTest, HighlighterTest, SolrIndexSplitterTest, HdfsLockFactoryTest,
TestQuerySenderListener, TestReplicationHandler, AnalyticsMergeStrategyTest,
TestFieldCollectionResource, TestNoOpRegenerator,
DistributedSpellCheckComponentTest, SolrCoreCheckLockOnStartupTest,
SpellPossibilityIteratorTest, QueryResultKeyTest, ReturnFieldsTest, TestSolrJ,
TestSort, ShardSplitTest, TestCloudManagedSchemaConcurrent, TestCodecSupport,
TestBinaryResponseWriter, TestManagedResourceStorage, TestClassNameShortening,
ZkControllerTest, DocExpirationUpdateProcessorFactoryTest, TestStressReorder,
MoreLikeThisHandlerTest, TestOverriddenPrefixQueryForCustomFieldType,
TestCursorMarkWithoutUniqueKey, TestSolrCoreProperties, TestRangeQuery,
TestFastLRUCache, DocValuesMultiTest, TestImplicitCoreProperties,
EchoParamsTest, TestSolrXmlPersistor, RequiredFieldsTest, SuggesterTSTTest,
CoreAdminRequestStatusTest, TestModifyConfFiles, TestSearcherReuse,
BlockDirectoryTest, TestConfigSets, DeleteShardTest, QueryEqualityTest,
TestNRTOpen, UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest,
FullSolrCloudDistribCmdsTest, TestRandomDVFaceting, AutoCommitTest,
TestFaceting, TermVectorComponentDistributedTest, TestReload, SimpleFacetsTest,
SolrCoreTest, SpellCheckComponentTest, TestGroupingSearch,
SolrCmdDistributorTest, BadIndexSchemaTest, TestFunctionQuery,
BasicFunctionalityTest, DirectUpdateHandlerTest, TestBadConfig,
OverseerCollectionProcessorTest, ShowFileRequestHandlerTest,
DistributedQueryElevationComponentTest, NoCacheHeaderTest, TestRemoteStreaming,
DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest,
TestOmitPositions, PathHierarchyTokenizerFactoryTest,
FastVectorHighlighterTest, LoggingHandlerTest, IndexSchemaTest,
UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest,
TestPartialUpdateDeduplication, PingRequestHandlerTest, TestComponentsName,
CoreContainerCoreInitFailuresTest, ResponseLogComponentTest, BadComponentTest,
TestSolrDeletionPolicy2, MultiTermTest, SampleTest, TestDocSet,
TestBinaryField, NumericFieldsTest, MinimalSchemaTest,
TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions,
ExternalFileFieldSortTest, TestPostingsSolrHighlighter,
DirectSolrConnectionTest, TestPhraseSuggestions, SynonymTokenizerTest,
TestDFRSimilarityFactory, TestPerFieldSimilarity,
TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory,
TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TimeZoneUtilsTest,
TestFastOutputStream, ScriptEngineTest, OpenExchangeRatesOrgProviderTest,
PluginInfoTest, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J0 in 40.18s, 1 test, 1 failure <<< FAILURES!
[...truncated 284 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:524: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:472: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1359:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961:
There were test failures: 437 suites, 1784 tests, 1 failure, 43 ignored (21
assumptions)
Total time: 108 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -client
-XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
