shashank created CAMEL-26784:
--------------------------------
Summary: camel-hazelcast - HazelcastRoutePolicy keeps the
leadership after its route is removed, and a follower consumes after its route
is suspended and resumed
Key: CAMEL-26784
URL: https://issues.apache.org/jira/browse/CAMEL-26784
Project: Camel
Issue Type: Bug
Components: camel-hazelcast
Reporter: shashank
Assignee: shashank
{{HazelcastRoutePolicy}} implements the leadership with a thread that takes the
Hazelcast lock and then sleeps while holding it. The thread is only interrupted
(and the lock released) when the policy itself stops, which happens when the
CamelContext stops. Two route lifecycle callbacks are not handled:
*A removed route keeps the leadership.* The policy does not override
{{onRemove}}. When the route is removed on the leader (with {{removeRoute}}, or
a route reload, which removes the route and adds it again), the policy keeps
the lock. The other nodes wait for the lock forever, so the route runs nowhere
in the cluster:
* removed on the leader: no other node takes over;
* reloaded on the leader with a new policy instance (for example a policy
created in the route builder): the new policy waits for the lock its
predecessor still holds, and so do the other nodes.
A route added again with the same policy instance still works, as the policy
stays the leader.
*A follower starts its consumer when its route is resumed.* On a node that is
not the leader the policy stops the consumer of the route in {{onStart}} and
starts it when the node takes the leadership. {{onSuspend}} removes the route
from the routes to start on takeover, but {{onResume}} is not overridden, so a
{{suspendRoute}} / {{resumeRoute}} on a follower (from JMX, the controlbus, a
scheduled route policy) resumes the consumer and nothing stops it again.
Depending on the consumer the follower then consumes next to the leader, or
(SEDA, whose resume does not restart the threads that a stop ended) its
consumer is reported started without consuming. The same happens when the
follower takes the leadership while its route is suspended: the takeover does
not start the suspended route, and the resume does not really start the
consumer the policy had stopped, so no node consumes.
Expected: a removed route gives up the leadership (when no other route uses the
policy), and a policy that gets a route again competes for it again; a resumed
route on a follower keeps its consumer stopped until the node takes the
leadership, as after a start, and a resumed route on a node that took the
leadership meanwhile consumes.
Note for the fix: interrupting the leadership thread is not enough on a client
instance ({{HazelcastRoutePolicy}} created with a {{HazelcastClient}}): a
client call made by an interrupted thread fails at once, so the {{remove}} and
{{unlock}} in the thread's {{finally}} block fail and the lock stays held until
the client disconnects. This already applies today to a policy that stops while
it holds the lock.
Suggested fix: count the routes of the policy; in {{onRemove}} of the last one
make the leadership thread release the lock and end without interrupting it
(for example it waits on a latch instead of sleeping), and start a new one in
{{onInit}} when a route is added again; release the lock with the interrupt
flag cleared; in {{onResume}} stop the consumer when the node is not the
leader, as {{onStart}} does, and restart a consumer the policy had stopped when
the node is now the leader.
Found with Specula (TLA+ model checked against traces of the real code) and
confirmed with tests on the real classes (two CamelContexts on an embedded
Hazelcast member, and a Hazelcast client).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
