andygrove opened a new pull request, #1777: URL: https://github.com/apache/datafusion-ballista/pull/1777
# Which issue does this PR close? Closes #1776. # Rationale for this change DataFusion 54 is approaching release. Ballista needs to be ready to track it so we can ship a matching release once 54.0.0 lands on crates.io. Picking the upgrade up early on a long-lived branch also surfaces the API churn before the rest of the ecosystem migrates. Because 54.0.0 has not been published yet, the workspace deps are pinned to a commit on `apache/datafusion` `branch-54` (currently `1321d60c`). This is a draft; we should rebase onto the released 54.0.0 (and switch the deps back to the published version string) before merging. # What changes are included in this PR? * Bump workspace deps to DataFusion 54 (git rev), `arrow`/`arrow-flight`/`parquet` 58.3, `object_store` 0.13.2, and `rustyline` 18.0.0 in `ballista-cli` to match `datafusion-cli`. * Drop the explicit `fn as_any(&self) -> &dyn Any` method from every `ExecutionPlan` / `TableProvider` impl in ballista. The trait method was removed in 54; downcasting now uses the new `dyn ExecutionPlan::is` / `downcast_ref` helpers (and the matching helpers on `dyn DataSource`, `dyn PhysicalExpr`). * Update `ExecutionPlan::partition_statistics` impls to return `Result<Arc<Statistics>>`. * Adapt to the new `PhysicalPlanDecodeContext` parameter on `parse_protobuf_partitioning` / `parse_protobuf_hash_partitioning`. * `BatchPartitioner::new_hash_partitioner` is now fallible; propagate the error. * `TaskContext::new` gained a `higher_order_functions` HashMap argument and `FunctionRegistry` gained `higher_order_function` / `higher_order_function_names`; wire both with empty defaults in `BallistaFunctionRegistry` and at every `TaskContext::new` call site. Verified locally: * `cargo check --workspace --all-targets --locked` * `cargo check -p ballista-scheduler -p ballista-executor -p ballista-core -p ballista --no-default-features --locked` * `cd ballista && cargo check --no-default-features --features standalone --locked` * `cargo clippy --all-targets --workspace --all-features -- -D warnings` * `cargo fmt --all -- --check` Test suite execution and the full workspace build are deferred to CI. # Are there any user-facing changes? Yes. Ballista will now require DataFusion 54.0.0. The minimum supported `rustyline` version for `ballista-cli` rises to 18.0.0. No public Ballista APIs are intentionally broken in this PR beyond the underlying DataFusion 54 churn that downstream embedders will already be tracking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
