andygrove commented on issue #1943:
URL:
https://github.com/apache/datafusion-ballista/issues/1943#issuecomment-4883648304
**Update: instrumented the repro (executor-side `h2=debug`) and root-caused
it. The failure is intermittent, and it's client-driven connection churn — not
a server-side reset.**
Re-ran `target_partitions=128` with `RUST_LOG=info,h2=debug` on both
executors (shuffle
fetches are executor→executor, so both sides matter). Notably, **this run of
128 passed
(22/22)** — the failure is a **timing race**, which is why the same config
fails some runs
and not others.
**Frame inventory for one run:**
| | count |
|---|---:|
| GOAWAY sent by **client** | **3,292** |
| GOAWAY sent by server | 348 |
| RST_STREAM (sent/recv) | **0** |
| SETTINGS frames | 18,660 |
| client-side `"Remote shuffle read fail, retry"` (all recovered) | 93 |
So under high `target_partitions` the **client opens and tears down
thousands of connections
per run** (the exclusive one-connection-per-fetch model), and `RST_STREAM:
0` rules out h2
rapid-reset. All the GOAWAYs are `NO_ERROR` (graceful). This churn is what
produces *both*
symptoms: with caching off it exhausts ephemeral ports (`EADDRNOTAVAIL`);
with caching on it
occasionally catches a fetch mid-stream (`BrokenPipe`).
**Two coupled defects in the shuffle client:**
1. **Connection returned to the pool before its stream is drained.** In
`fetch_partition_remote` (`shuffle_reader.rs`), the `PooledClient` guard
is a local that
drops the instant the function returns the stream — and
`PooledClient::drop`
(`client_pool.rs`) returns `self.client.clone()` to the pool. So a
connection is marked
idle/reusable/evictable **while its response body is still streaming**,
opening a
teardown-vs-read race (and connection aliasing via the clone).
2. **Retry doesn't cover the body phase.** `execute_do_get` (`client.rs`)
retries only the
`do_get` request and the first (schema) message, and retries on the
**same** connection. A
connection error while reading data batches (after the schema) is outside
the retry loop
entirely → hard `FetchFailed`. (The recoverable case is what produced the
93 successful
retries above; the mid-body case is the intermittent failure.)
**Fix direction (no buffering):**
- Tie the connection's lifetime to the **stream**: hold the `PooledClient`
until the returned
stream is fully consumed, so the pool can't reuse/evict/close a connection
mid-fetch.
- Reduce churn by reusing a few long-lived, multiplexed connections per peer
(h2 multiplexes
streams and auto-reconnects) instead of one connection per fetch — this
removes the
port-exhaustion mode outright.
I'm going to try a fix along these lines and verify it against the live
128/256 repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
