andygrove opened a new issue, #1950:
URL: https://github.com/apache/datafusion-ballista/issues/1950
## Describe the bug
Enabling adaptive query planning (`ballista.planner.adaptive.enabled=true`)
on a query
over **remote object-store data** (S3 in this case) fails the job with:
```
Internal error: No suitable object store found for s3://tpch/sf100/lineitem/.
See `RuntimeEnv::register_object_store`.
```
The **same query over the same data succeeds with adaptive planning
disabled** — the
non-adaptive execution path reads the S3 data fine. So the object store is
available to
normal execution but not to the adaptive-planning path.
## To Reproduce
Cluster: standalone scheduler + 2 executors (Ballista `main`), TPC-H SF100
stored in S3
(MinIO). The client registers the S3 object store on its `SessionContext`
(as the
`benchmarks` `tpch` binary does via its S3 support), and the query runs
against
`s3://tpch/sf100`.
Run the TPC-H benchmark with adaptive planning enabled:
```
tpch benchmark ballista \
--host <scheduler> --port 50050 \
--path s3://tpch/sf100 --format parquet --partitions 32 \
-c ballista.planner.adaptive.enabled=true \
-c ballista.planner.coalesce.enabled=true
```
The job fails immediately (0 queries complete):
```
[ERROR ballista_core::execution_plans::distributed_query] Job <id> failed:
DataFusion error: Internal error: No suitable object store found for
s3://tpch/sf100/lineitem/. See `RuntimeEnv::register_object_store`.
```
Removing the two `-c` overrides (adaptive disabled) → the query plans and
executes
normally and reads the S3 data.
## Expected behavior
Adaptive query planning should work with remote object stores. Whatever
`RuntimeEnv` the
adaptive re-planning runs against should have the same object store
registrations as the
normal execution path (or the adaptive rules should not require resolving
the object
store at plan time without it being registered).
## Additional context
- Both `ballista.planner.adaptive.enabled` and
`ballista.planner.coalesce.enabled` were
set; I have not isolated which rule triggers the object-store lookup. The
AQE
coalesce/size-estimation path seems the likely candidate (it needs scan
statistics),
but the master `adaptive.enabled` switch may be sufficient on its own —
worth
confirming.
- The failure is logged by the client-side `DistributedQueryExec`, but the
message is a
job failure reported back from the server side, so the missing
registration appears to
be on the scheduler/adaptive-planning path rather than the client (the
client session
has the store registered).
- The scheduler and executor pods have the S3 env vars (`AWS_ENDPOINT`,
`AWS_REGION`,
`AWS_ALLOW_HTTP`, credentials) available, but nothing in the
scheduler/executor
registers an object store from them by default — the store is only built
where a
session explicitly registers it.
- Environment: Ballista `main`, DataFusion 54, standalone deployment on
Kubernetes,
MinIO-backed S3.
This is likely a blocker for using adaptive planning with any non-local
object store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
