Vinod KC created SPARK-58094:
--------------------------------
Summary: Blocking RPCs hang forever when the connection dies
mid-wait (no gRPC keepalive / no client-side deadline)
Key: SPARK-58094
URL: https://issues.apache.org/jira/browse/SPARK-58094
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.2.0, 4.3.0
Reporter: Vinod KC
Attachments: spark-connect-hang-demo.zip
Any blocking call in Spark Connect can hang indefinitely and never throw if the
network path between client and server goes silently dark while the RPC is
waiting on a response (e.g. a NAT gateway, load balancer, or corporate proxy
evicting an idle connection without sending a TCP RST/FIN). This is a
transport-layer gap, not specific to any one API: Spark Connect configures no
gRPC/HTTP2 keepalive on either side, and the client's blocking read path has no
deadline by default. Any RPC whose response is delayed while its connection is
silently dropped is exposed — most visibly `StreamingQuery.awaitTermination()`
and other long-lived/idle-prone calls (long streaming queries, session status
polling, gaps between reattach calls), since they hold a connection open and
idle for long enough that a middlebox is likely to evict it — but the same
failure mode applies to a regular (non-streaming) query's RPC if its connection
happens to be silently dropped mid-wait; only the size of the exposure window
differs, not the mechanism.
*Impact:*
A Mission Critical case surfaced via StreamingQuery.awaitTermination(): a
streaming job stalled for hours after a single non-retryable source error.
Symptoms:
- StreamingQueryListener.onQueryTerminated() fired correctly with the terminal
exception.
- The client's blocked query.awaitTermination() call never returned and never
threw.
- The job run stayed "RUNNING" indefinitely; the driver process was healthy and
idle the whole time (no CPU/network activity, no errors logged).
*Root Cause:*
Two independent gaps combine to produce the hang, both at the transport/channel
level shared by all RPCs:
1. No gRPC/HTTP2 keepalive is configured on either the client channel
(SparkConnectClient.scala) or the server (SparkConnectService.scala's
NettyServerBuilder), so a silently-dropped connection is never detected.
2. The client's blocking read (ExecutePlanResponseReattachableIterator →
ArrayBlockingQueue.take()) has no deadline, and the server's polling loops
(e.g. handleStreamingAwaitTermination) only exit via
Context.current.isCancelled, a signal that itself depends on the transport
noticing the dead connection — which never happens without keepalive.
*Issue repro:*
Reproduced with two separate JVM processes (real Connect server + separate
client), using a `frozen TCP proxy` to simulate a silently-dropped connection
(no root/iptables needed).
Confirmed via thread dump that the client is parked forever in
ArrayBlockingQueue.take() while the server-side query has already terminated
independently. awaitTermination() was used as the concrete repro vehicle
because its long idle/blocking window makes the bug easy and reliable to
trigger, not because the bug is scoped to it.
Issue repro setup in zip file `spark-connect-hang-demo.zip`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
