Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4804/
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration
node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because
creating the ephemeral registration node in ZooKeeper failed
at
__randomizedtesting.SeedInfo.seed([17ADD52FB57E196F:13A55ADCA7DBF64E]:0)
at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:150)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
at
org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
at
org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
at
org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
at
org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
... 47 more
Build Log:
[...truncated 11628 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-17ADD52FB57E196F-001/init-core-data-001
[junit4] 2> 2447895 T7963 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
(false) and clientAuth (true)
[junit4] 2> 2447895 T7963 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 2447895 T7963 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 2447901 T7963 oas.SolrTestCaseJ4.setUp ###Starting
testPlaceholders
[junit4] 2> 2447902 T7963 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2447903 T7964 oasc.ZkTestServer$ZKServerMain.runFromConfig
Starting server
[junit4] 2> 2448003 T7963 oasc.ZkTestServer.run start zk server on
port:10592
[junit4] 2> 2448004 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2448007 T7970 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@4847e97d
name:ZooKeeperConnection Watcher:127.0.0.1:10592/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2448008 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2448009 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2448010 T7972 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@4feaa83c
name:ZooKeeperConnection Watcher:127.0.0.1:10592 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2448010 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2448013 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2448014 T7974 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@570573a1
name:ZooKeeperConnection Watcher:127.0.0.1:10592 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2448014 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2448015 T7963 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2448018 T7963 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2448020 T7963 oascc.SolrZkClient.makePath makePath:
/clusterstate.json
[junit4] 2> 2448021 T7963 oascc.SolrZkClient.makePath makePath:
/aliases.json
[junit4] 2> 2448022 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2448025 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2448026 T7976 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@18bbbf2 name:ZooKeeperConnection
Watcher:127.0.0.1:10592/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2448026 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2448027 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2448029 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node1
[junit4] 2> 2448031 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2448033 T7980 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@173af8f5
name:ZooKeeperConnection Watcher:127.0.0.1:10592/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2448033 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2448033 T7963 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 2448035 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect
[junit4] 2> 2448036 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/election
[junit4] 2> 2448037 T7963 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2448039 T7963 oasc.OverseerElectionContext.runLeaderProcess I
am going to be the leader 127.0.0.1:10592_solr
[junit4] 2> 2448039 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/leader
[junit4] 2> 2448040 T7963 oasc.Overseer.start Overseer
(id=92294846608441348-127.0.0.1:10592_solr-n_0000000000) starting
[junit4] 2> 2448042 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue
[junit4] 2> 2448043 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue-work
[junit4] 2> 2448045 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-failure
[junit4] 2> 2448047 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-running
[junit4] 2> 2448049 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-completed
[junit4] 2> 2448051 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-queue-work
[junit4] 2> 2448054 T7963 oasc.OverseerAutoReplicaFailoverThread.<init>
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2448054 T7982 oasc.OverseerCollectionProcessor.run Process
current queue of collection creations
[junit4] 2> 2448055 T7981 oasc.Overseer$ClusterStateUpdater.run Starting
to work on the main queue
[junit4] 2> 2448056 T7981 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=12 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"12",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 2448057 T7981
oasc.Overseer$ClusterStateUpdater.createCollection Create collection
collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6,
shard7, shard8, shard9, shard10, shard11, shard12]
[junit4] 2> 2448057 T7981 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard7
[junit4] 2> 2448059 T7984 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2448059 T7978 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2448059 T7977 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2448557 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard7/election
[junit4] 2> 2448563 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard7
[junit4] 2> 2448566 T7984 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2450057 T7982 oasc.OverseerCollectionProcessor.amILeader
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:516)
[junit4] 2> at
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:271)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2450057 T7982 oasc.OverseerCollectionProcessor.amILeader
According to ZK I (id=92294846608441348-127.0.0.1:10592_solr-n_0000000000) am
no longer a leader.
[junit4] 2> 2450068 T7981 oasc.Overseer$ClusterStateUpdater.run WARN Solr
cannot talk to ZK, exiting Overseer main queue loop
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /live_nodes
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:438)
[junit4] 2> at
org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:243)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:238)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2450069 T7981 oasc.Overseer$ClusterStateUpdater.run Overseer
Loop exiting : 127.0.0.1:10592_solr
[junit4] 2> 2450070 T7977 oascc.ZkStateReader$3.process WARN ZooKeeper
watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2450071 T7985
oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read
the data org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 2450073 T7963 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:10592 10592
[junit4] 2> 2450133 T7964 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:10592 10592
[junit4] 2> 2450135 T7963 oas.SolrTestCaseJ4.tearDown ###Ending
testPlaceholders
[junit4] 2> 2450135 T7963 oasc.Overseer.close Overseer
(id=92294846608441348-127.0.0.1:10592_solr-n_0000000000) closing
[junit4] 2> 2450143 T7963 oas.SolrTestCaseJ4.setUp ###Starting
testOverseerFailure
[junit4] 2> 2450144 T7963 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2450145 T7986 oasc.ZkTestServer$ZKServerMain.runFromConfig
Starting server
[junit4] 2> 2450245 T7963 oasc.ZkTestServer.run start zk server on
port:10598
[junit4] 2> 2450247 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2450251 T7992 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@485d3d86
name:ZooKeeperConnection Watcher:127.0.0.1:10598 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2450251 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2450254 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2450255 T7994 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@238d6071
name:ZooKeeperConnection Watcher:127.0.0.1:10598 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2450255 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2450256 T7963 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2450260 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2450262 T7996 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@52b09d88
name:ZooKeeperConnection Watcher:127.0.0.1:10598/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2450262 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2450262 T7963 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2450264 T7963 oascc.SolrZkClient.makePath makePath:
/clusterstate.json
[junit4] 2> 2450266 T7963 oascc.SolrZkClient.makePath makePath:
/aliases.json
[junit4] 2> 2450267 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2450268 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2450270 T7998 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@2eea420d
name:ZooKeeperConnection Watcher:127.0.0.1:10598/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2450270 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2450271 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2450272 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node1
[junit4] 2> 2450275 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2450276 T8002 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1d7c24e8
name:ZooKeeperConnection Watcher:127.0.0.1:10598/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2450276 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2450277 T7963 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 2450278 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect
[junit4] 2> 2450279 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/election
[junit4] 2> 2450280 T7963 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2450282 T7963 oasc.OverseerElectionContext.runLeaderProcess I
am going to be the leader 127.0.0.1:10598_solr
[junit4] 2> 2450282 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/leader
[junit4] 2> 2450283 T7963 oasc.Overseer.start Overseer
(id=92294846755373060-127.0.0.1:10598_solr-n_0000000000) starting
[junit4] 2> 2450285 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue
[junit4] 2> 2450286 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue-work
[junit4] 2> 2450288 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-failure
[junit4] 2> 2450290 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-running
[junit4] 2> 2450292 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-completed
[junit4] 2> 2450294 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-queue-work
[junit4] 2> 2450297 T7963 oasc.OverseerAutoReplicaFailoverThread.<init>
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2450297 T8004 oasc.OverseerCollectionProcessor.run Process
current queue of collection creations
[junit4] 2> 2450298 T8003 oasc.Overseer$ClusterStateUpdater.run Starting
to work on the main queue
[junit4] 2> 2451300 T8006 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2451302 T8003 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 2451302 T8003
oasc.Overseer$ClusterStateUpdater.createCollection Create collection
collection1 with shards [shard1]
[junit4] 2> 2451303 T8003 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard1
[junit4] 2> 2451305 T7999 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2451305 T8000 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2451800 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2451806 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2451809 T8006 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2451812 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2451813 T8003 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 2451813 T8003 oasc.Overseer$ClusterStateUpdater.updateState
shard=shard1 is already registered
[junit4] 2> 2451815 T8006 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2451815 T7999 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2451816 T8000 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2452299 T8004 oasc.OverseerCollectionProcessor.amILeader
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:516)
[junit4] 2> at
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:271)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2452300 T8004 oasc.OverseerCollectionProcessor.amILeader
According to ZK I (id=92294846755373060-127.0.0.1:10598_solr-n_0000000000) am
no longer a leader.
[junit4] 2> 2452870 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2452874 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2452876 T8008 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@4a410e13
name:ZooKeeperConnection Watcher:127.0.0.1:10598/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2452876 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2452876 T7963 oasc.Overseer.close Overseer
(id=92294846755373060-127.0.0.1:10598_solr-n_0000000000) closing
[junit4] 2> 2452877 T8003 oasc.Overseer$ClusterStateUpdater.run Overseer
Loop exiting : 127.0.0.1:10598_solr
[junit4] 2> 2452878 T7963 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 2452880 T8009
oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read
the data org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 2452881 T7963 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2452883 T7963 oasc.OverseerElectionContext.runLeaderProcess I
am going to be the leader 127.0.0.1:10598_solr
[junit4] 2> 2452883 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/leader
[junit4] 2> 2452884 T7963 oasc.Overseer.start Overseer
(id=92294846755373061-127.0.0.1:10598_solr-n_0000000001) starting
[junit4] 2> 2452890 T7963 oasc.OverseerAutoReplicaFailoverThread.<init>
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2452891 T8011 oasc.OverseerCollectionProcessor.run Process
current queue of collection creations
[junit4] 2> 2452891 T8010 oasc.Overseer$ClusterStateUpdater.run Starting
to work on the main queue
[junit4] 2> 2452893 T8010 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 2452893 T8010 oasc.Overseer$ClusterStateUpdater.updateState
shard=shard1 is already registered
[junit4] 2> 2452895 T8013 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2452895 T7999 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2452895 T8000 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2452947 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2452947 T8013 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2452950 T7963 oasc.LeaderElector.checkIfIamLeader WARN Failed
setting watch org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/collections/collection1/leader_elect/shard1/election/92294846755373059-node1_core1-n_0000000002
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:148)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] 2> at
org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
[junit4] 2> at
org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4] 2> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] 2> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4] 2> at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4] 2> at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
[junit4] 2> at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
[junit4] 2> at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4] 2> at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4] 2> at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4] 2> at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4] 2> at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2452952 T8014 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2452952 T8010 oasc.SolrException.log ERROR Problem cleaning up
collection in
zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException:
KeeperErrorCode = Directory not empty for
/collections/collection1/leader_elect/shard1/election
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:189)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:186)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:186)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:670)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:665)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:665)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:665)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1098)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:363)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:246)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2452951 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2452953 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2452955 T8013 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2452956 T8010 oasc.Overseer$ClusterStateUpdater.setShardLeader
ERROR Could not mark shard leader for non existing collection:collection1
[junit4] 2> 2453059 T8000 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2453059 T8014 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2453955 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2453956 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2454958 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2454959 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2455960 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2455961 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2456963 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2456963 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2457965 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2457966 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2458968 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2458968 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2459971 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2459971 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2460974 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2460974 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2461977 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2461977 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2462980 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2462980 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2463983 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2463983 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2464986 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2464986 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2465988 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2465989 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2466991 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2466992 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2467993 T7999 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2467994 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2467996 T7999 oasc.LeaderElector$ElectionWatcher.process WARN
org.apache.solr.common.SolrException: Could not register as the leader because
creating the ephemeral registration node in ZooKeeper failed
[junit4] 2> at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:150)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:209)
[junit4] 2> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] 2> at
java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /collections/collection1/leaders/shard1
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
[junit4] 2> at
org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
[junit4] 2> at
org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] 2> at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
[junit4] 2> ... 10 more
[junit4] 2>
[junit4] 2> 2469497 T7999 oascc.ZkStateReader$3.process WARN ZooKeeper
watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2469502 T7963 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:10598 10598
[junit4] 2> 2469845 T7986 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:10598 10598
[junit4] 2> 2469847 T7963 oas.SolrTestCaseJ4.tearDown ###Ending
testOverseerFailure
[junit4] 2> 2469848 T7963 oasc.Overseer.close Overseer
(id=92294846755373061-127.0.0.1:10598_solr-n_0000000001) closing
[junit4] 2> 2469848 T8010 oasc.Overseer$ClusterStateUpdater.run Overseer
Loop exiting : 127.0.0.1:10598_solr
[junit4] 2> 2469851 T8015
oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read
the data org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest
-Dtests.method=testOverseerFailure -Dtests.seed=17ADD52FB57E196F
-Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_BA_#Latn
-Dtests.timezone=America/Mazatlan -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 19.7s J1 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: Could not
register as the leader because creating the ephemeral registration node in
ZooKeeper failed
[junit4] > at
__randomizedtesting.SeedInfo.seed([17ADD52FB57E196F:13A55ADCA7DBF64E]:0)
[junit4] > at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:150)
[junit4] > at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] > at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] > at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
[junit4] > at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] > at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] > at
org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:157)
[junit4] > at
org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:662)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > Caused by:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /collections/collection1/leaders/shard1
[junit4] > at
org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] > at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:462)
[junit4] > at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] > at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:459)
[junit4] > at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:416)
[junit4] > at
org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:403)
[junit4] > at
org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:142)
[junit4] > at
org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] > at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:137)
[junit4] > ... 47 more
[junit4] 2> 2469873 T7963 oas.SolrTestCaseJ4.setUp ###Starting
testDoubleAssignment
[junit4] 2> 2469874 T7963 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2469875 T8016 oasc.ZkTestServer$ZKServerMain.runFromConfig
Starting server
[junit4] 2> 2469975 T7963 oasc.ZkTestServer.run start zk server on
port:33599
[junit4] 2> 2469976 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2469979 T8022 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@71e68447
name:ZooKeeperConnection Watcher:127.0.0.1:33599/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2469980 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2469981 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2469982 T8024 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@7417f8c7
name:ZooKeeperConnection Watcher:127.0.0.1:33599 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2469982 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2469985 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2469986 T8026 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@34f5484b
name:ZooKeeperConnection Watcher:127.0.0.1:33599 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2469986 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2469986 T7963 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2469990 T7963 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2469992 T7963 oascc.SolrZkClient.makePath makePath:
/clusterstate.json
[junit4] 2> 2469993 T7963 oascc.SolrZkClient.makePath makePath:
/aliases.json
[junit4] 2> 2469994 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2469996 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2469997 T8028 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@6da527f0
name:ZooKeeperConnection Watcher:127.0.0.1:33599/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2469997 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2469999 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2470000 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node1
[junit4] 2> 2470004 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2470005 T8032 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@7855add9
name:ZooKeeperConnection Watcher:127.0.0.1:33599/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2470005 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2470006 T7963 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 2470007 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect
[junit4] 2> 2470008 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/election
[junit4] 2> 2470009 T7963 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2470011 T7963 oasc.OverseerElectionContext.runLeaderProcess I
am going to be the leader 127.0.0.1:33599_solr
[junit4] 2> 2470011 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/leader
[junit4] 2> 2470012 T7963 oasc.Overseer.start Overseer
(id=92294848048398340-127.0.0.1:33599_solr-n_0000000000) starting
[junit4] 2> 2470013 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue
[junit4] 2> 2470015 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue-work
[junit4] 2> 2470017 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-failure
[junit4] 2> 2470018 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-running
[junit4] 2> 2470020 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-completed
[junit4] 2> 2470022 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-queue-work
[junit4] 2> 2470025 T7963 oasc.OverseerAutoReplicaFailoverThread.<init>
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2470026 T8034 oasc.OverseerCollectionProcessor.run Process
current queue of collection creations
[junit4] 2> 2470026 T8033 oasc.Overseer$ClusterStateUpdater.run Starting
to work on the main queue
[junit4] 2> 2470028 T8033 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 2470028 T8033
oasc.Overseer$ClusterStateUpdater.createCollection Create collection
collection1 with shards [shard1]
[junit4] 2> 2470028 T8033 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard1
[junit4] 2> 2470030 T8036 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2470030 T8029 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2470030 T8030 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2470527 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2470533 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2470536 T8036 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2470642 T8030 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 2472037 T8029 oascc.ZkStateReader$3.process WARN ZooKeeper
watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2472039 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2472040 T8038 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@7e167cf4
name:ZooKeeperConnection Watcher:127.0.0.1:33599/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2472041 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2472043 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2472044 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node1
[junit4] 2> 2472047 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leader_elect/shard1/election
[junit4] 2> 2472047 T8036 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2472049 T8033 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 2472049 T8033 oasc.Overseer$ClusterStateUpdater.updateState
shard=shard1 is already registered
[junit4] 2> 2472051 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection1/leaders/shard1
[junit4] 2> 2472053 T8036 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2472156 T8039 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2472156 T8030 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2473659 T8033 oasc.Overseer$ClusterStateUpdater.run WARN Solr
cannot talk to ZK, exiting Overseer main queue loop
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /overseer/queue-work
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:290)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:287)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:287)
[junit4] 2> at
org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:99)
[junit4] 2> at
org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:194)
[junit4] 2> at
org.apache.solr.cloud.DistributedQueue.poll(DistributedQueue.java:607)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:269)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 2473659 T8033 oasc.Overseer$ClusterStateUpdater.run Overseer
Loop exiting : 127.0.0.1:33599_solr
[junit4] 2> 2473659 T8039 oascc.ZkStateReader$3.process WARN ZooKeeper
watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2473662 T8040
oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read
the data org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 2473664 T7963 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:33599 33599
[junit4] 2> 2473859 T8016 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:33599 33599
[junit4] 2> 2473860 T7963 oas.SolrTestCaseJ4.tearDown ###Ending
testDoubleAssignment
[junit4] 2> 2473861 T7963 oasc.Overseer.close Overseer
(id=92294848048398340-127.0.0.1:33599_solr-n_0000000000) closing
[junit4] 2> 2473868 T7963 oas.SolrTestCaseJ4.setUp ###Starting
testShardAssignmentBigger
[junit4] 2> 2473870 T7963 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2473871 T8041 oasc.ZkTestServer$ZKServerMain.runFromConfig
Starting server
[junit4] 2> 2473971 T7963 oasc.ZkTestServer.run start zk server on
port:21165
[junit4] 2> 2473972 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2473975 T8047 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@5f53203b
name:ZooKeeperConnection Watcher:127.0.0.1:21165 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2473976 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2473978 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2473980 T8049 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@767606ca
name:ZooKeeperConnection Watcher:127.0.0.1:21165 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2473980 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2473980 T7963 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2473984 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2473985 T8051 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@12d7cb03
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2473985 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2473986 T7963 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2473988 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2473989 T8053 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@e9cdfc2 name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2473990 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2473990 T7963 oasu.UpdateShardHandler.<init> Creating
UpdateShardHandler HTTP client with params: retry=false
[junit4] 2> 2473991 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect
[junit4] 2> 2473993 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/election
[junit4] 2> 2473994 T7963 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2473995 T7963 oasc.OverseerElectionContext.runLeaderProcess I
am going to be the leader 127.0.0.1:21165_solr
[junit4] 2> 2473996 T7963 oascc.SolrZkClient.makePath makePath:
/overseer_elect/leader
[junit4] 2> 2473997 T7963 oasc.Overseer.start Overseer
(id=92294848310280195-127.0.0.1:21165_solr-n_0000000000) starting
[junit4] 2> 2473998 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue
[junit4] 2> 2474000 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/queue-work
[junit4] 2> 2474002 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-failure
[junit4] 2> 2474004 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-running
[junit4] 2> 2474005 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-map-completed
[junit4] 2> 2474007 T7963 oascc.SolrZkClient.makePath makePath:
/overseer/collection-queue-work
[junit4] 2> 2474010 T7963 oasc.OverseerAutoReplicaFailoverThread.<init>
Starting OverseerAutoReplicaFailoverThread
autoReplicaFailoverWorkLoopDelay=10000
autoReplicaFailoverWaitAfterExpiration=30000
autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2474011 T8055 oasc.OverseerCollectionProcessor.run Process
current queue of collection creations
[junit4] 2> 2474011 T7963 oascc.SolrZkClient.makePath makePath:
/clusterstate.json
[junit4] 2> 2474012 T8054 oasc.Overseer$ClusterStateUpdater.run Starting
to work on the main queue
[junit4] 2> 2474013 T7963 oascc.SolrZkClient.makePath makePath:
/aliases.json
[junit4] 2> 2474014 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474016 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474017 T8058 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@7988fbde
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474017 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474018 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474020 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node0
[junit4] 2> 2474022 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474023 T8062 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@531d2e name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474024 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474025 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474026 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node1
[junit4] 2> 2474029 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474030 T8065 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@6e213f4e
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474030 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474031 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474033 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node2
[junit4] 2> 2474035 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474036 T8068 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@3794ec5c
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474036 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474038 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474039 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node3
[junit4] 2> 2474042 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474043 T8071 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@3fbf50f name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474043 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474044 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474046 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node4
[junit4] 2> 2474048 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474049 T8074 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@b7938ff name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474050 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474051 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474052 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node5
[junit4] 2> 2474055 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474056 T8077 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@6b3dbf49
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474056 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474057 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474059 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node6
[junit4] 2> 2474061 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474062 T8080 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@3e53523e
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474063 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474064 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474065 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node7
[junit4] 2> 2474068 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474069 T8083 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@35eafe52
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474069 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474071 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474072 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node8
[junit4] 2> 2474075 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474076 T8086 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@2f611303
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474077 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474078 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474080 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node9
[junit4] 2> 2474083 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474084 T8089 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@2c84b568
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474085 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474086 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474087 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node10
[junit4] 2> 2474090 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474091 T8092 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@756336ec
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474092 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474093 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474094 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node11
[junit4] 2> 2474097 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474098 T8095 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@6f0822cc
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474099 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474100 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474101 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node12
[junit4] 2> 2474105 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474106 T8098 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@5f10679e
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474106 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474107 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474109 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node13
[junit4] 2> 2474112 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474113 T8101 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@a4ac013 name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474114 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474115 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474116 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node14
[junit4] 2> 2474119 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474120 T8104 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@73241a6b
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474120 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474122 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474123 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node15
[junit4] 2> 2474126 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474127 T8107 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@31b1edcf
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474128 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474129 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474130 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node16
[junit4] 2> 2474133 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474134 T8110 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@3a04062a
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474134 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474137 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474138 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node17
[junit4] 2> 2474141 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474142 T8113 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@2fa8271b
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474142 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474143 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474144 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node18
[junit4] 2> 2474147 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474148 T8116 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@11f8c0c name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474149 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474149 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474151 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node19
[junit4] 2> 2474154 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474155 T8119 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@5d7f8f13
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474155 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474156 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474157 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node20
[junit4] 2> 2474160 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474162 T8122 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@7d2b7c8a
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474162 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474163 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474164 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node21
[junit4] 2> 2474167 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474168 T8125 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@2ebe2be6
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474169 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474169 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474171 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node22
[junit4] 2> 2474174 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474175 T8128 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@1eb3d266
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474175 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474176 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474177 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node23
[junit4] 2> 2474180 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474181 T8131 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@66c405a name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474182 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474183 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474184 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node24
[junit4] 2> 2474187 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474188 T8134 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@53ea7009
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474188 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474189 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474191 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node25
[junit4] 2> 2474194 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474195 T8137 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@226b2b23
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474195 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474196 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474197 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node26
[junit4] 2> 2474201 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474202 T8140 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@6fbcb3a name:ZooKeeperConnection
Watcher:127.0.0.1:21165/solr got event WatchedEvent state:SyncConnected
type:None path:null path:null type:None
[junit4] 2> 2474202 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474203 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474204 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node27
[junit4] 2> 2474208 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474209 T8143 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@2fb451b2
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474209 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474210 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474211 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node28
[junit4] 2> 2474215 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client to connect to ZooKeeper
[junit4] 2> 2474216 T8146 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@68b3abfc
name:ZooKeeperConnection Watcher:127.0.0.1:21165/solr got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2474216 T7963 oascc.ConnectionManager.waitForConnected Client
is connected to ZooKeeper
[junit4] 2> 2474217 T7963
oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state
from ZooKeeper...
[junit4] 2> 2474218 T7963 oascc.SolrZkClient.makePath makePath:
/live_nodes/node29
[junit4] 2> 2474222 T7963 oascc.ConnectionManager.waitForConnected Waiting
for client
[...truncated too long message...]
ase_url":"http://127.0.0.1/solr/"}
[junit4] 2> 2524503 T8652 oasc.Overseer$ClusterStateUpdater.updateState
Collection already exists with numShards=3
[junit4] 2> 2524503 T8652 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard3
[junit4] 2> 2524505 T8659 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2524505 T8658 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2524505 T8657 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2525001 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection2/leader_elect/shard3/election
[junit4] 2> 2525005 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection2/leaders/shard3
[junit4] 2> 2525008 T8659 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2525011 T8652 oasc.Overseer$ClusterStateUpdater.updateState
Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection2",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 2525012 T8652 oasc.Overseer$ClusterStateUpdater.updateState
Collection already exists with numShards=3
[junit4] 2> 2525012 T8652 oasc.Overseer$ClusterStateUpdater.updateState
Assigning new node to shard shard=shard2
[junit4] 2> 2525013 T8659 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2525014 T8657 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2525014 T8658 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2525509 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection2/leader_elect/shard2/election
[junit4] 2> 2525514 T7963 oascc.SolrZkClient.makePath makePath:
/collections/collection2/leaders/shard2
[junit4] 2> 2525517 T8659 oasc.DistributedQueue$LatchChildWatcher.process
LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
NodeChildrenChanged
[junit4] 2> 2525621 T8658 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2525621 T8657 oascc.ZkStateReader$2.process A cluster state
change: WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2527171 T8657 oascc.ZkStateReader$3.process WARN ZooKeeper
watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2527175 T7963 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:16672 16672
[junit4] 2> 2527234 T8639 oasc.ZkTestServer.send4LetterWord connecting to
127.0.0.1:16672 16672
[junit4] 2> 2527236 T7963 oas.SolrTestCaseJ4.tearDown ###Ending
testBadQueueItem
[junit4] 2> 2527236 T7963 oasc.Overseer.close Overseer
(id=92294851475013635-127.0.0.1:16672_solr-n_0000000000) closing
[junit4] 2> 2527237 T8652 oasc.Overseer$ClusterStateUpdater.run Overseer
Loop exiting : 127.0.0.1:16672_solr
[junit4] 2> 2527237 T8660
oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read
the data org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at
org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:307)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:304)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:321)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89)
[junit4] 2> at
org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:293)
[junit4] 2>
[junit4] 2> 2530239 T7963 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-17ADD52FB57E196F-001
[junit4] 2> NOTE: test params are: codec=Lucene49, sim=DefaultSimilarity,
locale=sr_BA_#Latn, timezone=America/Mazatlan
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60
(64-bit)/cpus=16,threads=1,free=298973712,total=519045120
[junit4] 2> NOTE: All tests run in this JVM: [TestBadConfig,
CursorMarkTest, ShardRoutingCustomTest, TestDefaultSearchFieldResource,
OutputWriterTest, TestSolrQueryParser, LukeRequestHandlerTest,
TestAddFieldRealTimeGet, SampleTest, RequestHandlersTest, NumericFieldsTest,
SliceStateTest, TestStandardQParsers, AlternateDirectoryTest,
RAMDirectoryFactoryTest, CollectionsAPIDistributedZkTest, ClusterStateTest,
SOLR749Test, TestLFUCache, TestDocBasedVersionConstraints,
AbstractAnalyticsFacetTest, CoreAdminRequestStatusTest, AutoCommitTest,
TestInitQParser, TestReplicationHandlerBackup, TestDistribDocBasedVersion,
TestFunctionQuery, CoreContainerCoreInitFailuresTest, ConvertedLegacyTest,
TestXIncludeConfig, TestSchemaSimilarityResource, CustomCollectionTest,
TestCloudManagedSchemaConcurrent, TestReloadAndDeleteDocs,
TestIntervalFaceting, TestManagedStopFilterFactory, TestLeaderElectionZkExpiry,
ResourceLoaderTest, TestQueryTypes, TestCollectionAPI,
TestAtomicUpdateErrorCases, DocValuesTest,
DistributedQueryComponentCustomSortTest, TestExceedMaxTermLength,
TestJmxMonitoredMap, AnalysisErrorHandlingTest, SoftAutoCommitTest,
TestSolrQueryParserDefaultOperatorResource, ChaosMonkeySafeLeaderTest,
DistributedDebugComponentTest, TestCSVResponseWriter, TestWriterPerf,
SolrCoreTest, TestHighlightDedupGrouping, RemoteQueryErrorTest,
CoreMergeIndexesAdminHandlerTest, TestFiltering,
DistributedFacetPivotSmallTest, LoggingHandlerTest, TestManagedResource,
ShardSplitTest, TestHighFrequencyDictionaryFactory, TestRealTimeGet,
TestSolrQueryParserResource, SuggestComponentTest, TestDistributedGrouping,
URLClassifyProcessorTest, QueryParsingTest, TestRecoveryHdfs, CircularListTest,
BasicDistributedZk2Test, TestMissingGroups, TestQueryUtils, DebugComponentTest,
HdfsCollectionsAPIDistributedZkTest, FunctionTest, SolrInfoMBeanTest,
PolyFieldTest, SuggesterFSTTest, TestCharFilters, TestSolrXmlPersistence,
SpatialFilterTest, TestInfoStreamLogging, TestValueSourceCache,
LeaderElectionIntegrationTest, SolrPluginUtilsTest, CopyFieldTest,
TestNonNRTOpen, TestBM25SimilarityFactory, MultiThreadedOCPTest,
TestPHPSerializedResponseWriter, DistributedFacetPivotLongTailTest,
TestElisionMultitermQuery, OverseerRolesTest, TestUpdate,
TestIndexingPerformance, ChangedSchemaMergeTest,
DistributedQueryElevationComponentTest, ScriptEngineTest,
TestClassNameShortening, DateFieldTest, TestExtendedDismaxParser,
UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest]
[junit4] Completed on J1 in 82.38s, 9 tests, 1 error <<< FAILURES!
[...truncated 668 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:474:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:454:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:45:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:37:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:189:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:496:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1350:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:965:
There were test failures: 430 suites, 1782 tests, 1 error, 54 ignored (20
assumptions)
Total time: 115 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-Tests-trunk-Java7 #4803
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 184776407 bytes
Compression is 0.0%
Took 36 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
