[
https://issues.apache.org/jira/browse/IGNITE-29251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138392#comment-18138392
]
Mathias Roeppischer commented on IGNITE-29251:
----------------------------------------------
Proposed fix is available in [PR
#13791|https://github.com/apache/ignite/pull/13791].
The changes address the issues identified during the investigation:
* NIO session creation futures are now completed exceptionally when channel
registration fails.
* A single deadline covers the complete connection establishment process.
* Threads waiting for an ongoing connection attempt are released when their
reservation timeout expires or the communication SPI stops.
* A race between client creation and SPI shutdown is handled to prevent clients
from being published after shutdown.
The PR also adds regression tests for these scenarios, including SSL coverage.
The exact trigger of the original production incident could not be conclusively
established. The changes address the identified failure modes that can lead to
indefinitely blocked connection establishment or reservation waits.
> Communication connection establishment and client reservation can leave
> threads waiting indefinitely
> ----------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29251
> URL: https://issues.apache.org/jira/browse/IGNITE-29251
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 2.18
> Environment: Apache Ignite 2.18.0
> Java 25.0.4.1+1
> Linux 6.8.0-146-generic amd64
> Reporter: Mathias Roeppischer
> Assignee: Mathias Roeppischer
> Priority: Major
> Fix For: 2.20
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Description
> During a production incident two Ignite nodes simultaneously attempted to
> re-establish their communication connection. The connection recovery did not
> complete successfully on one side, and application threads subsequently
> accumulated while waiting for communication clients. The discovery
> communication remained operational.
> Available logs showed that an outgoing connection was accepted by the remote
> node, but the initiating node did not log successful establishment of that
> outgoing connection. No subsequent handshake timeout, connection failure,
> disconnect, send failure, or successful establishment was observed before the
> affected node was shut down manually. The striped-pool threads were
> eventually waiting in {{ConnectionClientPool.reserveClient()}} for
> connection-related futures.
> The exact point at which the original connection attempt stopped progressing
> could not be established conclusively from the available logs. In particular,
> the evidence does not prove whether the original incident was caused by a
> blocked TCP handshake, NIO session registration, or another point in the
> connection establishment sequence.
> The code analysis identified several independent conditions that could cause
> indefinite waits:
> * When NIO session registration fails with {{ClosedChannelException}} or
> {{IOException}}, the associated session-creation future is not completed
> exceptionally in all cases. A caller waiting for that future could therefore
> remain blocked.
> * The connection establishment timeout does not cover the complete sequence.
> In particular, the wait for NIO session registration is outside the TCP
> handshake timeout.
> * Threads waiting in {{ConnectionClientPool}} for a connection attempt could
> wait indefinitely if the shared connection future is not completed. Pending
> connection futures are also not all completed when the communication SPI
> stops.
> * A race between client creation and SPI shutdown could allow the creator to
> proceed without correctly accounting for shutdown winning completion of the
> shared future.
> h3. Expected behavior
> * Every failed NIO session registration must complete its associated future
> with an exception so that callers do not wait indefinitely.
> * The configured connection establishment timeout must cover socket
> connection, TCP handshake, and NIO session registration for an individual
> connection attempt.
> * Threads waiting for an already-running connection attempt must be bounded
> by the remaining client reservation timeout.
> * Stopping the communication SPI must complete pending client futures
> exceptionally and release threads waiting on them.
> * A client must only be returned when its creator successfully completes the
> shared future. If shutdown wins that completion race, the newly created
> client must be removed and closed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
