andygrove commented on issue #1943:
URL:
https://github.com/apache/datafusion-ballista/issues/1943#issuecomment-4883258222
**Update: verified the `--client-ttl` workaround — it does *not* resolve the
failure; it changes the failure mode.**
I re-ran with executors started with `--client-ttl=60` (connection caching
enabled), same
cluster (2 executors × 8 cores), same TPC-H SF100, `target_partitions=128`.
Result:
- The `EADDRNOTAVAIL` / "Cannot assign requested address" port-exhaustion
error is **gone** —
so caching does prevent the new-connection-per-fetch storm.
- But the query **still fails at the same point** (~stage 3, 2/22 queries),
now with a
different error — a reused pooled connection breaking mid-fetch:
```
h2 protocol error: error reading a body from connection
hyper::Error(Body, Io(Custom { kind: BrokenPipe, error: "stream closed
because of a broken pipe" }))
```
Both executors stayed healthy throughout (**0 restarts, no OOM /
termination**), so this is not
an executor crash — the pooled connection itself is breaking under the
high-partition shuffle
load, and the fetch is not retried on a fresh connection.
So high-`target_partitions` shuffle is unstable in **both** modes:
| shuffle client caching | failure at target_partitions=128 |
|---|---|
| off (default, `--client-ttl 0`) | ephemeral-port exhaustion
(`EADDRNOTAVAIL`) |
| on (`--client-ttl 60`) | `h2` broken-pipe on connection reuse |
(`target_partitions` 16 / 32 / 64 still complete fine in both modes.)
**Refined take / possible fixes:**
- Enabling the client pool by default is necessary but **not sufficient** on
its own.
- In `fetch_partition_remote` (`shuffle_reader.rs`), a failed pooled fetch
calls
`pooled.discard()` but does **not** retry on a fresh connection — so a
single stale/broken
pooled connection fails the whole query. Retrying the fetch with a new
connection when the
pooled one is broken would likely make this robust.
- It's also worth understanding *why* the connection is closed under load as
partition count
scales (h2 flow-control / max-concurrent-streams, server-side idle close,
etc.).
Environment: Ballista `main` (DataFusion 54), executors 8 cores each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
