Xuanwo opened a new pull request, #25194: URL: https://github.com/apache/datafusion/pull/25194
## Which issue does this PR close? Related to #14854 and #25144. This PoC does not close either discussion. ## Rationale for this change **This is a PoC to demonstrate my initial ideas, and the API is not final.** The main goal is to bring file I/O calls currently scattered throughout DataFusion behind one common entry point, so applications can register a storage backend once and use it across the existing file APIs. The API largely carries over the operations DataFusion already uses. It is a starting point for discussing the dependency boundary and integration model, with substantial room for future optimization and API changes. I would especially appreciate feedback on whether this direction provides the right place to integrate existing `object_store` clients, OpenDAL operators, and custom storage implementations. ## What changes are included in this PR? - Introduce a backend-independent `datafusion-storage` crate with `Storage` for namespace operations, `FileReader` for reads, and Tokio `AsyncWrite` for outputs. Shared file metadata and paths no longer depend on SDK types. - Route listing, schema inference, built-in file scans, `COPY TO`, and file table `INSERT` through the same registration and storage binding. Applications use `register_storage` once and continue using the ordinary format APIs. - Provide separate `object_store` and OpenDAL adapters. Existing SDK clients retain their configuration and middleware; backend features control adapter assembly instead of adding conditional branches throughout file operations. - Allow an OpenDAL-backed file stack without an `object_store` normal dependency, while preserving native operations such as batched ranges, suffix reads, delimiter listing, and backend writer buffering. - Update the affected CLI, examples, tests, and serialization call sites, and add integration and dependency checks. The [storage README](https://github.com/Xuanwo/datafusion/blob/xuanwo/storage-poc/datafusion/storage/README.md) describes the interfaces, registration model, and current limits. In this PoC, Avro and sequential Arrow IPC buffer complete inputs, including local files, after removing the SDK-specific local payload path. ## What is the testing strategy for this PR? Local validation completed: - Extended workspace tests: 11,385 Rust tests passed across 76 test targets, and all 513 SQL logic test files completed. - Storage and adapter tests, plus seven OpenDAL-only integration tests covering the normal format APIs, writes, manifest inputs, custom Parquet readers, registration replacement, and query contexts. The corresponding normal dependency graph contains no `object_store` package. - Existing CSV, JSON, and Parquet request snapshots retain their pre-refactor expectations. Additional adapter tests cover native read requests and writer buffer thresholds in `datafusion/core/tests/datasource/object_store_access.rs`. - Standalone CLI tests: 70 passed. - Formatting, whole-workspace/all-target/all-feature Clippy with warnings denied, documentation checks, and the repository lint suite. These tests cover local and in-memory backends and mocked services; they do not constitute cloud-service qualification. No performance improvement is claimed. ## Are there any user-facing changes? Yes. This PoC makes breaking changes to registration and file I/O extension APIs, including replacing `register_object_store` with `register_storage` and removing SDK types from shared file interfaces. **This PoC does not address compatibility or migration.** If this direction proves viable, I will separately design a roadmap for migrating the community to the new API. The changes here are intended to make the initial idea concrete and reviewable, not to establish the final API or migration plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
