Jerry Chung created SOLR-18558:
----------------------------------
Summary: Stale leader election znode left after DELETEREPLICA
blocks the remaining replica from becoming leader; registration fails with
"Could not get leader props"
Key: SOLR-18558
URL: https://issues.apache.org/jira/browse/SOLR-18558
Project: Solr
Issue Type: Bug
Components: Server, SolrCloud
Affects Versions: 10.0
Reporter: Jerry Chung
After a DELETEREPLICA on a shard, the election znode of the deleted replica was
left in /collections/<c>/leader_elect/<shard>/election, at the head of the
queue. The remaining replica joined the election behind it and never ran its
leader process. Registration of its core timed out after about 13 minutes, and
the shard stayed unloaded with no leader. Nothing logged that the replica was
waiting on another election node.
Environment
- Solr 10.0.0 (SolrCloud), ZooKeeper 3.9.4
- The shard has one replica, core_node93 (replica_n91)
- Node was restarted at 03:11 (clean shutdown) and the core loaded at 03:13:08
- DELETEREPLICA had earlier removed core_node94 from the same shard. [The node
hosting core_node94 was live at the time but possibly busy. Add the time and
the Overseer/collection-API log if available.]
Observed
1. ZkController.register → getLeader (ZkController.java:1419) polls
/collections/<c>/leaders/<shard>/leader once a second for leaderVoteWait +
600000 ms. At 03:26:09 it failed:
SolrException: Could not get leader props (ZkController.getLeaderProps:1715)
Caused by: KeeperException$NoNodeException: NoNode for
/collections/<c>/leaders/shard22/leader
SolrException: Error getting leader from zk for shard shard22
(ZkController.getLeader:1661)
ZkContainer: Exception registering core <c>_shard22_replica_n91
2. Election queue:
$ solr zk ls /collections/<c>/leader_elect/shard22_v10/election
144115188964327726-core_node94-n_0000000001
216172782927479120-core_node93-n_0000000005
core_node94 no longer exists in the clusterstate. Its node is ahead of
core_node93.
3. Terms also hold stale entries for the deleted replica:
{"core_node94":7, "core_node93":8, "core_node94_recovering":1}
These don't block core_node93: it has the max term and no _recovering
marker, so canBecomeLeader is true.
4. Between 03:13:08 and the timeout, the registration thread logged only
ShardLeaderElectionContextBase make sure parent is created
.../leaders/shard22_v10. There were no Enough replicas found, I may be the new
leader, Our node is no longer in line to be leader or Will not register as
leader lines. I found no ZK session expiry, disconnect or reconnect in the log
from 03:11 to 03:40.
Why the replica waits silently
In LeaderElector.checkIfIamLeader (line 125 onward), a replica that isn't first
in line only sets a watch on the node below it. The only log message there is
at DEBUG ("Watching path ... to know if I could be the leader"). A stale
head-of-queue node therefore blocks the shard with no visible warning.
The stale node is ephemeral, so some ZK session was still holding it. [Add the
owner of session 144115188964327726, converted to hex, from the ZK dump
command.]
Suspected cause (not confirmed or reproduced)
DELETEREPLICA cleans up terms and the election node only when UNLOAD reaches
the replica's node. unregister is the only place that does it
(ZkController.java:1964-1981). The paths below can leave the election node
behind, particularly if the node is slow or busy:
- DeleteReplicaCmd.java:311-332: UNLOAD is sent only if the node is live. If
the node is too slow, waitForCoreNodeGone times out after 30s and
deleteCoreNode removes the replica from the clusterstate anyway. The core may
still be loading, registering or electing on that node. A non-live node skips
UNLOAD entirely, so terms and election nodes are left behind.
- ShardLeaderElectionContext.runLeaderProcess (line 110): if (core == null) \{
return; } ("shutdown or removed") returns without calling cancelElection(). A
node that was already joined stays in the queue until the JVM's session ends.
- ZkController.unregister vs register: if unregister runs before the
registration thread calls joinElection, it finds no context and cancels
nothing. The registration thread then creates the election znode and context
(ZkController.java:1743-1749), and nothing cancels them afterward.
- ShardLeaderElectionContext.rejoinLeaderElection (line 503): it checks only
cc.isShutDown(), not whether the context is still registered. After unregister
has cancelled and removed the context, it can rejoin and leave an election node
that nothing owns.
Expected
- After DELETEREPLICA completes, no election znode or terms entry remains for
the deleted replica.
- A replica waiting behind another election node logs it at WARN, naming the
node it waits on.
- A shard whose only live replica is behind a stale election node recovers on
its own.
Impact
The shard is unloaded and has no leader, so it can't serve queries or index.
The registration thread blocks for 13 minutes (with default leaderVoteWait),
then fails.
Workaround
1. Delete the stale election znode: solr zk rm
/collections/<c>/leader_elect/<shard>/election/<stale-node> (add -z <zkHost> if
needed).
2. Reload the core or restart the node. The registration thread has already
given up.
3. Optionally remove the stale entries from terms/<shard>.
Proposed fixes (for discussion)
1. In runLeaderProcess, call cancelElection() before returning when the core is
null.
2. After joinElection in register, re-check that the replica still exists in
the clusterstate. Cancel and bail out if it doesn't.
3. In rejoinLeaderElection, skip the rejoin if the context is no longer in
electionContexts.
4. In checkIfIamLeader, log at WARN when waiting on a node whose replica isn't
in the clusterstate.
5. When DELETEREPLICA targets a non-live node, remove the replica's terms
entries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
