andygrove opened a new pull request, #27337: URL: https://github.com/apache/datafusion/pull/27337
## Which issue does this PR close? - Part of #27103. ## Rationale for this change As agreed in #27103, `datafusion-spark` is moving to the new [apache/datafusion-spark](https://github.com/apache/datafusion-spark) repository, along with its git history. `datafusion-spark` 56.0.0 is the last release from this repository (`branch-56` keeps the crate), and 57.0.0 will be released from the new one. This stays a draft until the code has been imported into apache/datafusion-spark. Open PRs that touch these files have a comment pointing to #27103. ## What changes are included in this PR? - Removes the crate (`datafusion/spark`), its sqllogictest files (`datafusion/sqllogictest/test_files/spark`) and the `audit-datafusion-spark-expression` agent skill, all of which move to the new repository. - `datafusion-cli`: removes the `--spark` flag and its test. Without the crate, the flag could only switch the SQL dialect, which `SET datafusion.sql_parser.dialect = 'spark'` already does. The CLI builds its context with `SessionContext::new_with_config_rt` again, as it did before #24179. - `datafusion-sqllogictest`: removes the Spark session setup for files under `spark/`. The Spark result formatting in the `DataFusion` engine stays, so the new repository can reuse the published engine. - CI: removes the `cargo check datafusion-spark features` job, its required check in `.asf.yaml`, its entries in `ci/scripts/check_rust_workflow_skips.py`, and the labeler rule. - Docs: points the Spark functions page and the contributor how-to at the new repository, removes `--spark` from the CLI usage page and the crate from the release publish steps, and adds a 57.0.0 upgrade note. `SparkOptions` (`datafusion.spark.*`) stays in `datafusion-common`, because `datafusion-spark` reads it. Moving it into the crate as a `ConfigExtension` can be a follow-up. ## What is the testing strategy for this PR? This PR removes code and adds no tests. I ran these locally with stable Rust 1.97, because the pinned 1.99.1 isn't available on my machine: - `cargo check --workspace --all-targets` - both clippy passes from `ci/scripts/rust_clippy.sh` with `-D warnings`. I allowed `clippy::from_iter_instead_of_collect`, `unfulfilled_lint_expectations` and `unknown_lints`, which only fire on 1.97 and only in code this PR doesn't touch. - `cargo fmt`, taplo, typos, prettier, the generated and examples docs checks, and the `.asf.yaml` and workflow-skip validators - the extended test suite from `CLAUDE.md`: 13,332 passed, 0 failed, including all 304 sqllogictest files - `cargo test --profile ci -p datafusion-cli`: 115 passed, 0 failed ## Are there any user-facing changes? Yes: - `datafusion-cli --spark` is removed. - `datafusion-spark` is published from apache/datafusion-spark starting with 57.0.0. The crate name doesn't change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
