andygrove opened a new pull request, #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173
## Summary
At SF1000 several TPC-H plans (Q11, Q21, Q22) encode above the 16 MiB
default gRPC message-size ceiling and fail with:
```
Status { code: OutOfRange, message: "Error, decoded message length too large:
found ~22 MB, the limit is: 16777216 bytes" }
```
Raising the runtime flags on scheduler/executor pods
(`--grpc-server-max-*-message-size=...`) only lifts the server-side *receive*
limits. Several paths still use `GrpcClientConfig::default()` (16 MiB)
unconditionally — most notably the scheduler-to-executor gRPC client in
`ExecutorManager::new`:
https://github.com/apache/datafusion-ballista/blob/main/ballista/scheduler/src/state/executor_manager.rs#L88
So task-assignment RPCs with large plans reject even when the servers have
been raised, unless the user knows to also configure a config-producer
override. Same story in `flight_proxy_service.rs` and the executor's client
pool `Default` lookups.
This PR bumps every default that fed a 16 MiB ceiling to **128 MiB**.
Context: I hit this in
[#2168](https://github.com/apache/datafusion-ballista/issues/2168) while
running SF1000 TPC-H and got stuck for a while before realizing the
scheduler-side flags don't cover this path.
## What changed
- `ballista/core/src/config.rs` — `BALLISTA_CLIENT_GRPC_MAX_MESSAGE_SIZE`
default (the SessionConfig-driven client path used by `distributed_query`).
- `ballista/core/src/utils.rs` — `GrpcClientConfig::default()`
`max_message_size` (used by scheduler-to-executor, flight proxy, executor
client pool default lookups).
- `ballista/scheduler/src/config.rs` and `ballista/executor/src/config.rs` —
both CLI flags (`--grpc-server-max-{decoding,encoding}-message-size`) and the
matching struct defaults.
- `ballista/executor/src/executor_process.rs` — matching
`ExecutorProcessConfig` defaults.
- Help strings updated.
The value is a **ceiling, not a preallocation**, so bumping it has no
runtime cost for messages below the old limit.
## New test
Adds a round-trip test that `ballista.client.grpc_max_message_size`, set on
a `SessionConfig` via `options_mut().set(...)`, reaches
`BallistaConfig::grpc_client_max_message_size()` unchanged. That path
(options.set → ExtensionOptions → get_usize_setting) had no existing coverage,
and it's the one Python clients rely on via their `cluster_config` overrides.
## Test plan
- [x] `cargo test --workspace --lib` — all green (all pre-existing suites +
2 new/updated).
- [x] `cargo fmt --all`.
- [x] `cargo clippy --workspace --all-targets -- -D warnings` clean.
- [ ] Reviewer sanity-check: any other places I missed that hard-code 16 MiB
in a *default*? (Non-default hard-coded values in examples/ were left alone.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
