[
https://issues.apache.org/jira/browse/IGNITE-29251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138393#comment-18138393
]
Mathias Roeppischer commented on IGNITE-29251:
----------------------------------------------
[~zstan]: Please review my changes.
> Communication connection establishment and client reservation can leave
> threads waiting indefinitely
> ----------------------------------------------------------------------------------------------------
>
> Key: IGNITE-29251
> URL: https://issues.apache.org/jira/browse/IGNITE-29251
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 2.18
> Environment: Apache Ignite 2.18.0
> Java 25.0.4.1+1
> Linux 6.8.0-146-generic amd64
> Reporter: Mathias Roeppischer
> Assignee: Mathias Roeppischer
> Priority: Major
> Fix For: 2.20
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Description
> During a production incident two Ignite nodes simultaneously attempted to
> re-establish their communication connection. The connection recovery did not
> complete successfully on one side, and application threads subsequently
> accumulated while waiting for communication clients. The discovery
> communication remained operational.
> Available logs showed that an outgoing connection was accepted by the remote
> node, but the initiating node did not log successful establishment of that
> outgoing connection. No subsequent handshake timeout, connection failure,
> disconnect, send failure, or successful establishment was observed before the
> affected node was shut down manually. The striped-pool threads were
> eventually waiting in {{ConnectionClientPool.reserveClient()}} for
> connection-related futures.
> The exact point at which the original connection attempt stopped progressing
> could not be established conclusively from the available logs. In particular,
> the evidence does not prove whether the original incident was caused by a
> blocked TCP handshake, NIO session registration, or another point in the
> connection establishment sequence.
> The code analysis identified several independent conditions that could cause
> indefinite waits:
> * When NIO session registration fails with {{ClosedChannelException}} or
> {{IOException}}, the associated session-creation future is not completed
> exceptionally in all cases. A caller waiting for that future could therefore
> remain blocked.
> * The connection establishment timeout does not cover the complete sequence.
> In particular, the wait for NIO session registration is outside the TCP
> handshake timeout.
> * Threads waiting in {{ConnectionClientPool}} for a connection attempt could
> wait indefinitely if the shared connection future is not completed. Pending
> connection futures are also not all completed when the communication SPI
> stops.
> * A race between client creation and SPI shutdown could allow the creator to
> proceed without correctly accounting for shutdown winning completion of the
> shared future.
> h3. Expected behavior
> * Every failed NIO session registration must complete its associated future
> with an exception so that callers do not wait indefinitely.
> * The configured connection establishment timeout must cover socket
> connection, TCP handshake, and NIO session registration for an individual
> connection attempt.
> * Threads waiting for an already-running connection attempt must be bounded
> by the remaining client reservation timeout.
> * Stopping the communication SPI must complete pending client futures
> exceptionally and release threads waiting on them.
> * A client must only be returned when its creator successfully completes the
> shared future. If shutdown wins that completion race, the newly created
> client must be removed and closed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
