[
https://issues.apache.org/jira/browse/CAMEL-25613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25613:
--------------------------------
Fix Version/s: 4.24.0
> camel-jmx - consumer with reconnectOnConnectionFailure=true closes and
> replaces a working connection every reconnectDelay for ever after a lost
> connection
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25613
> URL: https://issues.apache.org/jira/browse/CAMEL-25613
> Project: Camel
> Issue Type: Bug
> Components: camel-jmx
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> All of this is about a remote JMX consumer with
> {{reconnectOnConnectionFailure=true}}. The option is off by default; it is
> the option one sets so that the consumer survives an outage of the monitored
> JVM, and the defect follows every such outage. {{JMXConsumer}}, lines at main
> a7169d7c6ab6:
> * {{initNetworkConnection}} (139-154) closes the previous connector (142),
> connects (150), adds the connection listener (151) and stores the connection
> id (152);
> * {{ConnectionNotificationListener.handleNotification}} (190-207) ignores a
> notification whose connection id is not the stored one (193) and calls
> {{scheduleReconnect()}} for {{NOTIFS_LOST}}, {{CLOSED}} and {{FAILED}}
> (196-201);
> * {{scheduleReconnect}} (213-226) schedules one task per call; the task calls
> {{initNetworkConnection()}} and {{addNotificationListener()}}, and any
> exception schedules it again (220).
> h3. 1. One lost connection schedules two reconnects, and the replacement
> never stops
> The RMI connector of the JDK reports a connection that can no longer be used
> with {{jmx.remote.connection.failed}} and then, from the close that follows,
> with {{jmx.remote.connection.closed}}, both with the id of that connection
> (observed with JDK 21;
> {{RMIConnector.RMIClientCommunicatorAdmin.gotIOException}} sends the first
> and calls {{close(true)}}, which sends the second). Both pass the id check
> (193), so two reconnect tasks are scheduled.
> When the server is back, the first task connects and subscribes. The second
> task then closes that working connector (142). Closing a connector whose
> connection works sends {{jmx.remote.connection.closed}} with the id of that
> connection to the connection listener, synchronously, inside {{close()}}; the
> id is still the stored one (it is replaced at 152), so the listener schedules
> a third task; and so on. Result: after an outage the consumer closes and
> replaces a working connection every {{reconnectDelay}} (10 seconds by
> default) for as long as the route runs, with a WARN "Lost JMX connection"
> each time, and notifications that the MBean sends between a close and the
> next subscription are lost.
> Observed in a local run with main's {{JMXConsumer}}, a JMX server in a second
> JVM that was killed and started again on the same port, and
> {{reconnectDelay=1}}: the connector delivered {{failed}} and {{closed}} with
> the stored id from one thread, the consumer logged "Lost JMX connection" and
> "Delaying JMX consumer reconnection" twice, and the new server accepted 16
> connections from the consumer in the first 8 seconds (two every second).
> The "closed" notification of its own connector keeps the replacement going in
> two more cases:
> * after one {{NOTIFS_LOST}} (the server dropped notifications for this
> client; the connection itself is fine);
> * when an attempt connected but {{addNotificationListener}} failed, typically
> with {{InstanceNotFoundException}}: the JMX connector of a restarting
> application is usually reachable before the application has registered its
> MBeans.
> h3. 2. The consumer does not follow the connection id after the connector
> reconnected by itself
> When the server object is gone but the server still answers
> ({{NoSuchObjectException}}, for example a connector server that was stopped
> and started again inside a JVM that keeps running), the RMI connector tries
> once to re-establish its connection. If that works it sends
> {{jmx.remote.connection.opened}} with a new connection id, and it adds again
> only the listeners whose MBean exists at that moment (read in the JDK
> sources: {{RMIClientCommunicatorAdmin.doStart}} and
> {{reconnectNotificationListeners}}). The consumer ignores that notification
> and keeps the old id, so:
> * every later {{FAILED}} / {{CLOSED}} / {{NOTIFS_LOST}} carries the new id
> and is ignored (193): the consumer never reconnects again, although
> {{reconnectOnConnectionFailure=true}}, and nothing is logged;
> * when the MBean was not registered at the moment of the reconnection, the
> consumer has a connection and no subscription, receives nothing, and logs
> nothing.
> h3. 3. A notification after stop reconnects the stopped consumer
> The lost-connection branch of the listener (196-201) does not check whether
> the consumer still runs. {{doStop}} shuts the scheduled executor down and
> forgets it (266-269); a notification that is handled after that makes
> {{getExecutor()}} (232-238) create a new executor, and the stopped consumer
> connects again.
> h3. Reproduction
> New {{JMXConsumerReconnectTest}} (RMI registry and connector server in the
> test JVM, set up like the existing {{JMXRobustRemoteConnectionTest}};
> {{reconnectDelay=1}}; the connector server counts the client connections it
> accepts). On main all five tests fail:
> * {{FAILED}} and then {{CLOSED}} with the current connection id are given to
> the connection listener: one new connection is expected and then none for
> three seconds. On main: {{the consumer connected again although it was
> subscribed on a working connection ==> expected: <false> but was: <true>}};
> * a {{NOTIFS_LOST}} notification for the current connection: one new
> connection and then none. On main the same message;
> * the server is stopped and started again without the MBean, the test waits
> for two new connections, registers the MBean and waits for a notification;
> after that the connections must stop. On main: {{the consumer kept connecting
> although it was subscribed on a working connection}} (when the connector
> reported the loss) or {{the consumer did not keep trying to subscribe after
> the server came back without the MBean}} (when the connector reconnected by
> itself; which of the two happens depends on timing);
> * an {{OPENED}} notification with another connection id: the consumer must
> open a connection of its own again. On main: {{the consumer did not subscribe
> again after the connector had reconnected by itself ==> expected: <true> but
> was: <false>}};
> * the route is stopped and {{FAILED}} with the last connection id is given to
> the listener: no connection may follow. On main: {{the consumer connected
> again although it was stopped ==> expected: <false> but was: <true>}}.
> h3. Proposed fix
> * {{scheduleReconnect}} schedules at most one reconnect at a time: a flag is
> set with {{compareAndSet}} when a reconnect is scheduled, cleared when the
> task begins (so that a failure reported during the attempt schedules the next
> one) and reset in {{doStop}}. It schedules nothing when the consumer is not
> allowed to run, and the task ends at once in that case.
> * {{initNetworkConnection}} removes the connection listener from the previous
> connector before it closes it.
> * An {{OPENED}} notification with another id: the consumer stores the new id
> and, with {{reconnectOnConnectionFailure=true}}, schedules its normal
> reconnect, which subscribes again on a new connection (it cannot know whether
> the connector kept its listener). A WARN line is logged; with
> {{reconnectOnConnectionFailure=false}} only a WARN line that the consumer may
> not be subscribed anymore.
> * The listener ignores a lost connection when the consumer is not allowed to
> run.
> * The fields that the notification threads read are volatile.
> One consequence to know: when the connector re-established the connection by
> itself and all listeners were added again, main keeps receiving without a gap
> (and is deaf to later failures). With the fix the consumer replaces that
> connection once, {{reconnectDelay}} later; notifications that the MBean sends
> between the close and the new subscription are lost that one time.
> Not changed: with {{testConnectionOnStartup=false}} a delayed start that
> connects but cannot add the listener still logs "An unrecoverable exception
> has occurred" at ERROR and does not try again; {{doStop}} still fails when
> the server cannot be reached; a reconnect attempt that is already connecting
> when the consumer is stopped is not interrupted and may leave a connection
> open; the WARN "Lost JMX connection" is still logged for each of the two
> notifications.
> camel-jmx tests with the fix: 58 run, 0 failures, 0 skipped. With the fix and
> the same second-JVM run as above the new server accepted one connection and
> the next notification of the MBean was delivered. Not run: a JMX server on
> another host.
> How it was found: a sweep of the reconnect code of the components, then a
> small TLA+ model (server up or down, MBean registered, connector state,
> stored id, pending and running reconnect, the "failed" and "closed" pair,
> stop). For main TLC reports two pending reconnects after one lost connection
> (4 steps), a replaced connection without a cause, "nothing pending, server
> up, MBean registered, not subscribed", a run that never ends subscribed with
> nothing pending, and a reconnect scheduled after stop. With the fix the
> properties hold for three environment events and one stop (739 states). The
> model is a sketch; the tests are the evidence.
> Affected: main (a7169d7c6ab6; tested). camel-4.23.x, camel-4.22.x,
> camel-4.18.x and camel-4.14.x have the same id check, the same
> {{initNetworkConnection}} and the same {{scheduleReconnect}} (read, not run).
> Duplicate check (2026-10-13): JIRA text "jmx reconnectOnConnectionFailure"
> (CAMEL-5613), "JMXConsumer reconnect" (CAMEL-19527, tests only), "jmx
> consumer connection notification" (nothing related); GitHub pull requests
> "JMXConsumer", "camel-jmx reconnect": #2052 (2017, close the connector on
> shutdown), none for this.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
