[
https://issues.apache.org/jira/browse/CAMEL-26784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-26784:
--------------------------------
Fix Version/s: 4.24.0
> camel-hazelcast - HazelcastRoutePolicy keeps the leadership after its route
> is removed, and a follower consumes after its route is suspended and resumed
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26784
> URL: https://issues.apache.org/jira/browse/CAMEL-26784
> Project: Camel
> Issue Type: Bug
> Components: camel-hazelcast
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> {{HazelcastRoutePolicy}} implements the leadership with a thread that takes
> the Hazelcast lock and then sleeps while holding it. The thread is only
> interrupted (and the lock released) when the policy itself stops, which
> happens when the CamelContext stops. Two route lifecycle callbacks are not
> handled:
> *A removed route keeps the leadership.* The policy does not override
> {{onRemove}}. When the route is removed on the leader (with {{removeRoute}},
> or a route reload, which removes the route and adds it again), the policy
> keeps the lock. The other nodes wait for the lock forever, so the route runs
> nowhere in the cluster:
> * removed on the leader: no other node takes over;
> * reloaded on the leader with a new policy instance (for example a policy
> created in the route builder): the new policy waits for the lock its
> predecessor still holds, and so do the other nodes.
> A route added again with the same policy instance still works, as the policy
> stays the leader.
> *A follower starts its consumer when its route is resumed.* On a node that is
> not the leader the policy stops the consumer of the route in {{onStart}} and
> starts it when the node takes the leadership. {{onSuspend}} removes the route
> from the routes to start on takeover, but {{onResume}} is not overridden, so
> a {{suspendRoute}} / {{resumeRoute}} on a follower (from JMX, the controlbus,
> a scheduled route policy) resumes the consumer and nothing stops it again.
> Depending on the consumer the follower then consumes next to the leader, or
> (SEDA, whose resume does not restart the threads that a stop ended) its
> consumer is reported started without consuming. The same happens when the
> follower takes the leadership while its route is suspended: the takeover does
> not start the suspended route, and the resume does not really start the
> consumer the policy had stopped, so no node consumes.
> Expected: a removed route gives up the leadership (when no other route uses
> the policy), and a policy that gets a route again competes for it again; a
> resumed route on a follower keeps its consumer stopped until the node takes
> the leadership, as after a start, and a resumed route on a node that took the
> leadership meanwhile consumes.
> Note for the fix: interrupting the leadership thread is not enough on a
> client instance ({{HazelcastRoutePolicy}} created with a
> {{HazelcastClient}}): a client call made by an interrupted thread fails at
> once, so the {{remove}} and {{unlock}} in the thread's {{finally}} block fail
> and the lock stays held until the client disconnects. This already applies
> today to a policy that stops while it holds the lock.
> Suggested fix: count the routes of the policy; in {{onRemove}} of the last
> one make the leadership thread release the lock and end without interrupting
> it (for example it waits on a latch instead of sleeping), and start a new one
> in {{onInit}} when a route is added again; release the lock with the
> interrupt flag cleared; in {{onResume}} stop the consumer when the node is
> not the leader, as {{onStart}} does, and restart a consumer the policy had
> stopped when the node is now the leader.
> Found with Specula (TLA+ model checked against traces of the real code) and
> confirmed with tests on the real classes (two CamelContexts on an embedded
> Hazelcast member, and a Hazelcast client).
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
