shashank created CAMEL-25613:
--------------------------------
Summary: camel-jmx - consumer with
reconnectOnConnectionFailure=true closes and replaces a working connection
every reconnectDelay for ever after a lost connection
Key: CAMEL-25613
URL: https://issues.apache.org/jira/browse/CAMEL-25613
Project: Camel
Issue Type: Bug
Components: camel-jmx
Reporter: shashank
Assignee: shashank
All of this is about a remote JMX consumer with
{{reconnectOnConnectionFailure=true}}. The option is off by default; it is the
option one sets so that the consumer survives an outage of the monitored JVM,
and the defect follows every such outage. {{JMXConsumer}}, lines at main
a7169d7c6ab6:
* {{initNetworkConnection}} (139-154) closes the previous connector (142),
connects (150), adds the connection listener (151) and stores the connection id
(152);
* {{ConnectionNotificationListener.handleNotification}} (190-207) ignores a
notification whose connection id is not the stored one (193) and calls
{{scheduleReconnect()}} for {{NOTIFS_LOST}}, {{CLOSED}} and {{FAILED}}
(196-201);
* {{scheduleReconnect}} (213-226) schedules one task per call; the task calls
{{initNetworkConnection()}} and {{addNotificationListener()}}, and any
exception schedules it again (220).
h3. 1. One lost connection schedules two reconnects, and the replacement never
stops
The RMI connector of the JDK reports a connection that can no longer be used
with {{jmx.remote.connection.failed}} and then, from the close that follows,
with {{jmx.remote.connection.closed}}, both with the id of that connection
(observed with JDK 21;
{{RMIConnector.RMIClientCommunicatorAdmin.gotIOException}} sends the first and
calls {{close(true)}}, which sends the second). Both pass the id check (193),
so two reconnect tasks are scheduled.
When the server is back, the first task connects and subscribes. The second
task then closes that working connector (142). Closing a connector whose
connection works sends {{jmx.remote.connection.closed}} with the id of that
connection to the connection listener, synchronously, inside {{close()}}; the
id is still the stored one (it is replaced at 152), so the listener schedules a
third task; and so on. Result: after an outage the consumer closes and replaces
a working connection every {{reconnectDelay}} (10 seconds by default) for as
long as the route runs, with a WARN "Lost JMX connection" each time, and
notifications that the MBean sends between a close and the next subscription
are lost.
Observed in a local run with main's {{JMXConsumer}}, a JMX server in a second
JVM that was killed and started again on the same port, and
{{reconnectDelay=1}}: the connector delivered {{failed}} and {{closed}} with
the stored id from one thread, the consumer logged "Lost JMX connection" and
"Delaying JMX consumer reconnection" twice, and the new server accepted 16
connections from the consumer in the first 8 seconds (two every second).
The "closed" notification of its own connector keeps the replacement going in
two more cases:
* after one {{NOTIFS_LOST}} (the server dropped notifications for this client;
the connection itself is fine);
* when an attempt connected but {{addNotificationListener}} failed, typically
with {{InstanceNotFoundException}}: the JMX connector of a restarting
application is usually reachable before the application has registered its
MBeans.
h3. 2. The consumer does not follow the connection id after the connector
reconnected by itself
When the server object is gone but the server still answers
({{NoSuchObjectException}}, for example a connector server that was stopped and
started again inside a JVM that keeps running), the RMI connector tries once to
re-establish its connection. If that works it sends
{{jmx.remote.connection.opened}} with a new connection id, and it adds again
only the listeners whose MBean exists at that moment (read in the JDK sources:
{{RMIClientCommunicatorAdmin.doStart}} and {{reconnectNotificationListeners}}).
The consumer ignores that notification and keeps the old id, so:
* every later {{FAILED}} / {{CLOSED}} / {{NOTIFS_LOST}} carries the new id and
is ignored (193): the consumer never reconnects again, although
{{reconnectOnConnectionFailure=true}}, and nothing is logged;
* when the MBean was not registered at the moment of the reconnection, the
consumer has a connection and no subscription, receives nothing, and logs
nothing.
h3. 3. A notification after stop reconnects the stopped consumer
The lost-connection branch of the listener (196-201) does not check whether the
consumer still runs. {{doStop}} shuts the scheduled executor down and forgets
it (266-269); a notification that is handled after that makes {{getExecutor()}}
(232-238) create a new executor, and the stopped consumer connects again.
h3. Reproduction
New {{JMXConsumerReconnectTest}} (RMI registry and connector server in the test
JVM, set up like the existing {{JMXRobustRemoteConnectionTest}};
{{reconnectDelay=1}}; the connector server counts the client connections it
accepts). On main all five tests fail:
* {{FAILED}} and then {{CLOSED}} with the current connection id are given to
the connection listener: one new connection is expected and then none for three
seconds. On main: {{the consumer connected again although it was subscribed on
a working connection ==> expected: <false> but was: <true>}};
* a {{NOTIFS_LOST}} notification for the current connection: one new connection
and then none. On main the same message;
* the server is stopped and started again without the MBean, the test waits for
two new connections, registers the MBean and waits for a notification; after
that the connections must stop. On main: {{the consumer kept connecting
although it was subscribed on a working connection}} (when the connector
reported the loss) or {{the consumer did not keep trying to subscribe after the
server came back without the MBean}} (when the connector reconnected by itself;
which of the two happens depends on timing);
* an {{OPENED}} notification with another connection id: the consumer must open
a connection of its own again. On main: {{the consumer did not subscribe again
after the connector had reconnected by itself ==> expected: <true> but was:
<false>}};
* the route is stopped and {{FAILED}} with the last connection id is given to
the listener: no connection may follow. On main: {{the consumer connected again
although it was stopped ==> expected: <false> but was: <true>}}.
h3. Proposed fix
* {{scheduleReconnect}} schedules at most one reconnect at a time: a flag is
set with {{compareAndSet}} when a reconnect is scheduled, cleared when the task
begins (so that a failure reported during the attempt schedules the next one)
and reset in {{doStop}}. It schedules nothing when the consumer is not allowed
to run, and the task ends at once in that case.
* {{initNetworkConnection}} removes the connection listener from the previous
connector before it closes it.
* An {{OPENED}} notification with another id: the consumer stores the new id
and, with {{reconnectOnConnectionFailure=true}}, schedules its normal
reconnect, which subscribes again on a new connection (it cannot know whether
the connector kept its listener). A WARN line is logged; with
{{reconnectOnConnectionFailure=false}} only a WARN line that the consumer may
not be subscribed anymore.
* The listener ignores a lost connection when the consumer is not allowed to
run.
* The fields that the notification threads read are volatile.
One consequence to know: when the connector re-established the connection by
itself and all listeners were added again, main keeps receiving without a gap
(and is deaf to later failures). With the fix the consumer replaces that
connection once, {{reconnectDelay}} later; notifications that the MBean sends
between the close and the new subscription are lost that one time.
Not changed: with {{testConnectionOnStartup=false}} a delayed start that
connects but cannot add the listener still logs "An unrecoverable exception has
occurred" at ERROR and does not try again; {{doStop}} still fails when the
server cannot be reached; a reconnect attempt that is already connecting when
the consumer is stopped is not interrupted and may leave a connection open; the
WARN "Lost JMX connection" is still logged for each of the two notifications.
camel-jmx tests with the fix: 58 run, 0 failures, 0 skipped. With the fix and
the same second-JVM run as above the new server accepted one connection and the
next notification of the MBean was delivered. Not run: a JMX server on another
host.
How it was found: a sweep of the reconnect code of the components, then a small
TLA+ model (server up or down, MBean registered, connector state, stored id,
pending and running reconnect, the "failed" and "closed" pair, stop). For main
TLC reports two pending reconnects after one lost connection (4 steps), a
replaced connection without a cause, "nothing pending, server up, MBean
registered, not subscribed", a run that never ends subscribed with nothing
pending, and a reconnect scheduled after stop. With the fix the properties hold
for three environment events and one stop (739 states). The model is a sketch;
the tests are the evidence.
Affected: main (a7169d7c6ab6; tested). camel-4.23.x, camel-4.22.x, camel-4.18.x
and camel-4.14.x have the same id check, the same {{initNetworkConnection}} and
the same {{scheduleReconnect}} (read, not run).
Duplicate check (2026-10-13): JIRA text "jmx reconnectOnConnectionFailure"
(CAMEL-5613), "JMXConsumer reconnect" (CAMEL-19527, tests only), "jmx consumer
connection notification" (nothing related); GitHub pull requests "JMXConsumer",
"camel-jmx reconnect": #2052 (2017, close the connector on shutdown), none for
this.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
