Xuanwo opened a new pull request, #25144:
URL: https://github.com/apache/datafusion/pull/25144
## Which issue does this PR close?
No linked issue.
## Rationale for this change
Applications that provide their own `TableProvider` and `ExecutionPlan`
implementations currently depend on `object_store` even with DataFusion's
default features disabled. This prevents embedders from excluding an unused
storage implementation or avoiding a second `object_store` version alongside
their own storage layer.
## What changes are included in this PR?
Add a default-enabled `object_store` feature for built-in file sources and
sinks, listing tables, the runtime object store registry, and file caches.
Forward the feature through the existing execution dependency chain so direct
child-crate users retain storage support by default. No new DataFusion packages
or replacement storage abstractions are introduced.
Custom providers, in-memory tables, query planning and execution, memory
management, and local spill support remain available without this feature.
Logical-only Substrait conversion no longer pulls in `object_store`; built-in
Parquet, Avro, and physical Substrait retain their storage dependencies.
## What is the testing strategy for this PR?
`ci/scripts/check_object_store_features.py` creates an isolated downstream
consumer so workspace feature unification cannot hide a dependency regression.
It checks `cargo tree --target all --edges all` for 15 entry points, requiring
storage with defaults and no `object_store` package with defaults disabled.
The accompanying fixture implements a custom two-partition file reader and
runs projection, filtering, joins, aggregation, sorting, and limit through SQL.
It runs with storage disabled, explicitly restored, and enabled by default; the
storage-enabled cases also exercise built-in CSV reading, registry lookup, and
file cache configuration. The script is wired into the existing feature-check
CI job.
Local validation includes the downstream script, `./dev/rust_lint.sh`, and
`cargo clippy --all-targets --all-features -- -D warnings`.
## Are there any user-facing changes?
Default builds keep their existing storage APIs and concrete `object_store`
types, including direct child-crate consumers.
**API change for existing builds with default features disabled:** callers
using storage APIs must add `object_store` to their features, unless an enabled
feature such as `parquet` or `avro` already enables it. The same requirement
applies to child crates.
To opt out:
```toml
datafusion = { version = "55.0.0", default-features = false, features =
["sql"] }
```
To retain storage with defaults disabled:
```toml
datafusion = { version = "55.0.0", default-features = false, features =
["sql", "object_store"] }
```
The crate configuration guide documents migration and feature unification:
another dependency can re-enable storage. With storage disabled, built-in file
APIs are unavailable, file `COPY TO` fails explicitly, and file cache settings
are unavailable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
