shashank created CAMEL-26570:
--------------------------------
Summary: camel-milo - the loss report of a consumer started
without a connection runs on the request pool and can deadlock the connection
Key: CAMEL-26570
URL: https://issues.apache.org/jira/browse/CAMEL-26570
Project: Camel
Issue Type: Bug
Components: camel-milo
Reporter: shashank
Assignee: shashank
Found with a TLA+ model of the camel-milo client connection, then reproduced
with the embedded Milo server of the tests.
Since CAMEL-25835 a consumer that starts while its connection is not up (the
server cannot be reached, or Camel is rebuilding the connection) reports its
nodes with {{Bad_NoCommunication}}. {{SubscriptionManager.registerItems}} hands
that report to the {{MiloClient}} pool of the connection (CAMEL-25713), and
{{MiloClientConsumer}} runs the route synchronously on that thread. The same
pool completes the producer requests ({{outcomeOf}}: {{whenCompleteAsync(...,
executor)}}) and has as many threads as there are processors: one in a
container limited to one CPU.
When the route waits for a request of the same connection (an awaited write,
read or method call: {{defaultAwaitWrites=true}} or the {{CamelMiloAwait}}
header) that is sent once the connection is up, its outcome is queued behind
the thread that waits for it. With as many such consumers as the pool has
threads:
* their exchanges wait for ever, and so does every other awaited request of the
connection;
* the values stop for every consumer of the connection: the waiting report
holds the lock of its node ({{Subscription.report}} is synchronized), and the
delivery queue of the shared subscription waits behind the first value of that
node;
* stopping the context waits for its timeouts and for the interrupt of the
waiting threads.
The request reaches the connection after it is up when the error handler
delivers it again (while there is no connection it fails at once with "No
connected"), or when the route is slower than the connection attempt:
{code:java}
errorHandler(defaultErrorHandler().maximumRedeliveries(600).redeliveryDelay(100));
from("milo-client:opc.tcp://localhost:4840?node=RAW(ns=2;s=valve1)")
.setBody(constant(new Variant("ack")))
.to("milo-client:opc.tcp://localhost:4840?node=RAW(ns=2;s=valve1Ack)&defaultAwaitWrites=true");
{code}
Not affected: requests that are not awaited (the default), routes that do not
wait for a request after the connection came up, and virtual threads with more
than one processor.
Short trace (model, one thread):
{noformat}
reconnect thread: performConnect (outside the lock)
consumer: registerItems: connected == null -> executor.execute(report)
reconnect thread: putSubscriptions, connected = C1
executor thread: report -> route -> write on C1, waits for its outcome
Milo thread: answer -> whenCompleteAsync(executor): queued behind the
waiting thread
{noformat}
On main (1b12eea752e6), with one consumer route like the above per processor
and the server reachable a moment after the routes started, none of the writes
completed in 30 s; with one consumer fewer they all completed. A manager with a
one-thread request pool whose consumer writes from the report once the
connection is up times out the same way.
*Proposal*: deliver that report on Milo's shared executor
({{Stack.sharedExecutor()}}, the cached pool the clients deliver the values
on), not on the request pool. It keeps generation 0, so it is still dropped
once a connection has delivered a value. The model holds with this change.
Affects main only (4.24.0-SNAPSHOT: CAMEL-25835 and CAMEL-25713 are not
released).
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
