shashank created CAMEL-27296:
--------------------------------
Summary: camel-kubernetes - a stopped KubernetesClusterView keeps
reporting the leadership and keeps the lease, so no other pod can lead
Key: CAMEL-27296
URL: https://issues.apache.org/jira/browse/CAMEL-27296
Project: Camel
Issue Type: Bug
Components: camel-kubernetes
Reporter: shashank
Assignee: shashank
h3. What happens
{{KubernetesClusterView.doStop}} only stops the leadership controller (its
executors) and closes the client. It does not clear {{currentLeader}}, does not
fire leadership lost, and does not release the lease. A view is stopped when
its last user releases it ({{MasterConsumer}}, {{ClusteredRoutePolicy}}) or
when the cluster service stops. Then:
* the stopped view keeps reporting the pod as leader
({{getLocalMember().isLeader()}}, {{getLeader()}},
{{CamelClusterService.isLeader(namespace)}});
* the lease ({{Lease}} holderIdentity, or the leader key of the group in the
{{ConfigMap}}) still names the pod, and the pod is still ready, so the other
pods keep treating it as the valid leader: no pod can lead the group until this
pod becomes unready or goes away;
* when the view is used again (a clustered route added again), the restarted
view reports the old leadership at once, until its first successful lookup (for
ever if the API server cannot be reached), also while another pod is the leader.
{{FileLockClusterView.doStop}} releases its lock and fires leadership lost.
h3. How to reproduce
{{KubernetesClusterViewStopTest}} (camel-kubernetes, {{LockTestServer}} mock,
Lease and ConfigMap): release the view of the leader pod so it stops; the view
still says leader, the lease still names the pod, and the other pod never
becomes leader.
Found with a TLA+ model of the Kubernetes leadership state machine.
h3. Fix
{{KubernetesLeadershipController.stop}} waits for a running refresh (up to the
renew deadline); the view clears {{currentLeader}} and fires leadership lost if
the pod was the leader; then {{releaseLeadership}} reads the lease again and,
if it names the pod, removes the pod from it with
{{optimisticDeleteLeaderInfo}} (409 reads again, up to 3 tries; other failures
are logged at WARN and the lease is kept, as before). The release runs on its
own thread and the stop waits for it up to the renew deadline, as the
Kubernetes client tries a failed request again on its own (by default up to 10
times), which could otherwise hold the stop for minutes when the API server
cannot be reached. The lease is only released when the leadership had taken
effect (not while the pod waits in BECOMING_LEADER for the previous leader).
Events of a stopped view's controller are ignored. Upgrade guide entry for 4.24.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)