andygrove opened a new pull request, #4888: URL: https://github.com/apache/datafusion-comet/pull/4888
## Summary WIP draft to track the upgrade to DataFusion main (currently pinned to `63e25c15bc72e731c174ba03d1143b243cc1aaae`, dated 2026-07-10). Bumps arrow/parquet from 58.3.0 to 59.1.0 to match what DataFusion main uses. `object_store` stays at 0.13.2 (DF main is still on that). Not ready for merge — opened as a draft so CI runs against latest DF main and breaking changes are surfaced continuously. ## Changes so far - `native/Cargo.toml`: swap `datafusion`, `datafusion-datasource`, `datafusion-physical-expr-adapter`, `datafusion-spark` from crates.io `54.0.0` to `git = "https://github.com/apache/datafusion";, rev = "..."` - `native/core/Cargo.toml`: same treatment for the dev-dependency `datafusion-functions-nested` - Bump `arrow` and `parquet` workspace pins to `59.1.0` ## Known breakage (initial cargo check) `GroupsAccumulator::merge_batch` signature changed upstream — dropped the `opt_filter: Option<&BooleanArray>` parameter (now 3 args instead of 4). 15 compile errors across: - `spark-expr/src/agg_funcs/{avg,avg_decimal,correlation,covariance,percentile,stddev,sum_decimal,sum_int,variance}.rs` Also deprecation warnings on `MutableArrayData::extend`/`extend_nulls` (arrow 59 renamed to `try_extend`/`try_extend_nulls` returning errors instead of panicking) across several `spark-expr/src/array_funcs/*.rs` files. ## Test plan - [ ] `cd native && cargo check --workspace` compiles clean - [ ] `make test-rust` passes - [ ] `make test-jvm` passes on default profile - [ ] Address any additional breakage revealed as compile errors are worked through -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
