mustafa kamal created CAMEL-24457:
-------------------------------------
Summary: Camel Zookeeper Cluster Split brain issue when leader is
isolated- #25611
Key: CAMEL-24457
URL: https://issues.apache.org/jira/browse/CAMEL-24457
Project: Camel
Issue Type: Bug
Components: camel-zookeeper, camel-zookeeper-master
Affects Versions: 4.18.3
Environment: 3 Linux RHEL machines
All 3 has ZK on it
And 2 Has the the camel application in cluster mode Active/Passive
Reporter: mustafa kamal
------------------------------------------------------------------------------------------------------------------
Problem Description:
------------------------------------------------------------------------------------------------------------------
When the ZooKeeper cluster becomes unavailable, the Camel route managed by
"ClusteredRoutePolicy" never stops, even though the node has lost its ZooKeeper
session. Beyond the route not stopping, this also introduces a split-brain
scenario: if one node becomes isolated from ZooKeeper while others remain
connected, a new leader election runs on the healthy side and a second node
wins leadership and starts the same route — now two nodes are running the same
route simultaneously with no coordination, which can cause data corruption,
duplicate processing, or conflicting writes depending on what the route does.
------------------------------------------------------------------------------------------------------------------
Root Cause:
------------------------------------------------------------------------------------------------------------------
The root cause is a timing issue between two threads in
"ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()". When
ZooKeeper goes down, Curator's "ConnectionStateManager" thread fires a
"SUSPENDED"/"LOST" state change, which causes "LeaderSelectorListenerAdapter"
to interrupt the thread blocked inside "takeLeadership()". The "BlockingTask"
handles the interrupt correctly and exits its loop. Execution then reaches the
leadership-lost event at:
fireLeadershipChangedEvent(getLeader().orElse(null)); //
ZooKeeperClusterView.java line 155
This fires "ClusteredRoutePolicy.leadershipChanged()", which, regardless of the
argument passed, always calls back into:
setLeader(clusterView.getLocalMember().isLeader()); //
ClusteredRoutePolicy.java line 376
Which resolves to: leaderSelector.hasLeadership() //
CuratorLocalMember.isLeader(), line 162
This returns "true" at this point because Curator's guarantees that
"hasLeadership()" only becomes "false" after "takeLeadership()" returns to the
"LeaderSelector" internals. The event fires from inside "takeLeadership()", so
the answer is always "true", "ClusteredRoutePolicy" sees no leadership change,
and the route keeps running indefinitely.
------------------------------------------------------------------------------------------------------------------
Consequences:
------------------------------------------------------------------------------------------------------------------
The consequence is:
- The [isolated node] keeps running the route because "hasLeadership()" is
still "true" at the moment the event fires
- The [healthy side] elects a new leader, which also starts the same route
- Both nodes now process the same workload simultaneously with no mutual
exclusion — a classic [split-brain]
------------------------------------------------------------------------------------------------------------------
Solution:
------------------------------------------------------------------------------------------------------------------
Introduce a "volatile boolean leader" flag inside "ZooKeeperClusterView" that
is owned and controlled by the view itself, rather than delegating to
"leaderSelector.hasLeadership()".
Change "CuratorLocalMember.isLeader()" to return this flag instead.
In "takeLeadership()", set the flag to "true" before firing the
leadership-gained event, and in the "finally" block set it to "false" before
firing the leadership-lost event. This guarantees that when
"ClusteredRoutePolicy" calls back into "isLeader()" during the event, it reads
"false", which causes "stopManagedRoutes()" to be called and the route to stop
correctly before any other node can win the election and start it.
Additionally, "leaderSelector.autoRequeue()" should be called in "doStart()" so
that after losing leadership due to a ZooKeeper disconnect, the node
automatically re-enters the election when ZooKeeper reconnects and the route
can start again on whichever node wins.
- "ZooKeeperClusterView": add "volatile boolean leader" field
- "ZooKeeperClusterView.CuratorLocalMember.isLeader()": return "leader" flag
instead of "leaderSelector.hasLeadership()"
- "ZooKeeperClusterView.CamelLeaderElectionListener.takeLeadership()": set
"leader = true" on entry, wrap task in "try/finally", set "leader = false"
before firing the lost event
- "ZooKeeperClusterView.doStart()": add "leaderSelector.autoRequeue()" to
re-enter election after reconnect
------------------------------------------------------------------------------------------------------------------
How to reproduce:
------------------------------------------------------------------------------------------------------------------
1. Start two instances of a Camel application using "ZooKeeperClusterService"
with "ClusteredRouteController"
2. Confirm one instance is leader and its route is running
3. Isolate the leader node from all ZooKeeper nodes (e.g. firewall rules or
kill ZK nodes)
4. Observe on the healthy side: a new leader is elected and its route starts
5. Observe on the isolated node: the route never stops — both nodes are now
running the same route simultaneously (split-brain)
6. Expected: the isolated node's route stops as soon as its ZooKeeper session
is lost
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
