shashank created CAMEL-24951:
--------------------------------
Summary: Stopping a suspended SEDA route (or the CamelContext)
with queued messages always waits for the full shutdown timeout
Key: CAMEL-24951
URL: https://issues.apache.org/jira/browse/CAMEL-24951
Project: Camel
Issue Type: Bug
Components: camel-seda, camel-core
Reporter: shashank
A suspended {{SedaConsumer}} does not poll, but graceful shutdown waits for it
to drain its queue:
* {{DefaultShutdownStrategy}} defers the seda consumer ({{deferShutdown}}
returns true) and loops until inflight + {{getPendingExchangesSize()}} is 0.
For seda, {{getPendingExchangesSize()}} is the queue size
(DefaultShutdownStrategy.java:667-716, SedaConsumer.java:88-95).
* After that, {{SedaConsumer.prepareShutdown}} waits on the poll threads'
latch. The poll threads exit only when {{shutdownPending && queue.isEmpty()}},
including in the suspended branch (SedaConsumer.java:175-191, :117).
When the route is suspended and messages were sent to it in the meantime,
neither wait can finish, so every stop runs into the timeout (default 45 s):
* {{stopRoute(id)}} and {{camelContext.stop()}} take the full timeout and then
force the shutdown.
* {{stopRoute(id, timeout, unit, true)}} returns false and restarts the route.
Reproduced against 4.23.0-SNAPSHOT, shutdown timeout 5 s, route
{{from("seda:s")}} suspended with {{suspendRoute("s")}}:
{noformat}
stopRoute, route suspended, 0 queued : stop took 490 ms,
timeoutOccurred=false
stopRoute, route suspended, 3 queued : stop took 5002 ms,
timeoutOccurred=true, left in queue=2
context.stop, route suspended, 0 queued : stop took 507 ms,
timeoutOccurred=false
context.stop, route suspended, 3 queued : stop took 5005 ms,
timeoutOccurred=true
{noformat}
During the forced stop the suspended route still processed one of the queued
messages.
CAMEL-6390 fixed stopping a suspended seda route with an empty queue. The
queued-messages case is still open ({{SedaSuspendConsumerStopRouteTest}} stops
with an empty queue). Routes suspended by a {{RoutePolicy}} such as
{{ThrottlingInflightRoutePolicy}} end up in this state without user action, and
route policies are not invoked while Camel is stopping, so they cannot resume
the consumer either.
Proposed fix: a suspended consumer cannot drain, so do not wait for it.
* {{SedaConsumer.getPendingExchangesSize()}} returns 0 while the consumer is
suspending or suspended (after the {{purgeWhenStopping}} purge, which is
unchanged).
* The suspended branch of {{doRun}} breaks out when {{shutdownPending}} is set,
whether or not the queue is empty.
The queued messages stay in the queue (or are purged by {{purgeWhenStopping}})
and are processed if the route is started again while the queue still exists.
The stop completes in about one poll interval instead of the shutdown timeout,
and the suspended route no longer processes a message during the forced stop.
Design point for reviewers: the alternative is to drain, i.e. resume suspended
seda consumers when stopping them. That would process messages on a route the
user (or a {{RoutePolicy}}) suspended, and the route's own services are
suspended too, so the patch keeps the messages instead. Today's outcome after
the timeout is also "not drained", just 45 s later.
A PR with the fix follows, with regression test
{{SedaSuspendedRouteWithPendingStopTest}} ({{stopRoute}} with
{{abortAfterTimeout=true}}, and {{context.stop()}}), plus an upgrade-guide note.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
