Mathias Roeppischer created IGNITE-29251:
--------------------------------------------
Summary: Communication connection establishment and client
reservation can leave threads waiting indefinitely
Key: IGNITE-29251
URL: https://issues.apache.org/jira/browse/IGNITE-29251
Project: Ignite
Issue Type: Bug
Affects Versions: 2.18
Environment: Apache Ignite 2.18.0
Java 25.0.4.1+1
Linux 6.8.0-146-generic amd64
Reporter: Mathias Roeppischer
Assignee: Mathias Roeppischer
Fix For: 2.20
h3. Description
During a production incident two Ignite nodes simultaneously attempted to
re-establish their communication connection. The connection recovery did not
complete successfully on one side, and application threads subsequently
accumulated while waiting for communication clients. The discovery
communication remained operational.
Available logs showed that an outgoing connection was accepted by the remote
node, but the initiating node did not log successful establishment of that
outgoing connection. No subsequent handshake timeout, connection failure,
disconnect, send failure, or successful establishment was observed before the
affected node was shut down manually. The striped-pool threads were eventually
waiting in {{ConnectionClientPool.reserveClient()}} for connection-related
futures.
The exact point at which the original connection attempt stopped progressing
could not be established conclusively from the available logs. In particular,
the evidence does not prove whether the original incident was caused by a
blocked TCP handshake, NIO session registration, or another point in the
connection establishment sequence.
The code analysis identified several independent conditions that could cause
indefinite waits:
* When NIO session registration fails with {{ClosedChannelException}} or
{{IOException}}, the associated session-creation future is not completed
exceptionally in all cases. A caller waiting for that future could therefore
remain blocked.
* The connection establishment timeout does not cover the complete sequence. In
particular, the wait for NIO session registration is outside the TCP handshake
timeout.
* Threads waiting in {{ConnectionClientPool}} for a connection attempt could
wait indefinitely if the shared connection future is not completed. Pending
connection futures are also not all completed when the communication SPI stops.
* A race between client creation and SPI shutdown could allow the creator to
proceed without correctly accounting for shutdown winning completion of the
shared future.
h3. Expected behavior
* Every failed NIO session registration must complete its associated future
with an exception so that callers do not wait indefinitely.
* The configured connection establishment timeout must cover socket connection,
TCP handshake, and NIO session registration for an individual connection
attempt.
* Threads waiting for an already-running connection attempt must be bounded by
the remaining client reservation timeout.
* Stopping the communication SPI must complete pending client futures
exceptionally and release threads waiting on them.
* A client must only be returned when its creator successfully completes the
shared future. If shutdown wins that completion race, the newly created client
must be removed and closed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
