Merged the fix for CI, I can create RC3 today, the question should we wait for updated arrow in this release or wait for 53.1.0? I would incline for 53.1.0 if there is no stoppers
On 2026/03/17 14:48:41 Matt Butrovich wrote: > Just to clarify: the panic that I opened a DataFusion issue for in the hash > join exec is not caused by the BatchCoalescer, but an overflow. > > That said, I think this arrow-rs BatchCoalescer debug_assert needs a patch > release before we can ship DF 53 since Comet is having issues with that too. > > On Tue, Mar 17, 2026 at 10:43 AM Andrew Lamb <[email protected]> wrote: > > > The only way really to fix the panic (in debug builds) will be a new > > version of arrow-rs. I will work on that this week. > > > > Should we hold the DataFusion 53 release for that? > > > > On Tue, Mar 17, 2026 at 10:28 AM Matt Butrovich <[email protected]> > > wrote: > > > > > -1 (non-binding) > > > > > > CometFuzzTestSuite "join" test causes a panic in the hash join operator: > > > https://github.com/apache/datafusion/issues/20995 > > > > > > On Tue, Mar 17, 2026 at 8:54 AM Tim Saucer <[email protected]> wrote: > > > > > > > +1 binding > > > > > > > > Also verified with `datafusion-python`. > > > > > > > > On Tue, Mar 17, 2026 at 7:42 AM Andrew Lamb <[email protected]> > > > > wrote: > > > > > > > > > +1 (binding) > > > > > I verified on Mac M3 > > > > > > > > > > I believe the verification error is due to a known bug in arrow-rs[1] > > > > > > > > > > Here is a proposed fix to the verification script:[2]. I used the > > > > modified > > > > > script and was able to successfully verify the release candidate > > > > > > > > > > I will backport the changes to branch-53 for future reference too > > > > > > > > > > Andrew > > > > > > > > > > [1]: https://github.com/apache/datafusion/issues/20689 > > > > > [2]: https://github.com/apache/datafusion/pull/20987 > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2026 at 3:11 AM Martin Grigorov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Same tests fail here too. > > > > > > > > > > > > On Tue, Mar 17, 2026 at 5:42 AM Kevin Liu <[email protected]> > > > > wrote: > > > > > > > > > > > > > I ran the release script on Mac twice. Getting some failed tests > > > both > > > > > > > times. > > > > > > > > > > > > > > Here's the stack trace for the first two: > > > > > > > ``` > > > > > > > failures: > > > > > > > > > > > > > > ---- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > joins::hash_join::exec::tests::join_full_multi_batch::batch_size_4_2::use_perfect_hash_join_as_possible_1_true > > > > > > > stdout ---- > > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'joins::hash_join::exec::tests::join_full_multi_batch::batch_size_4_2::use_perfect_hash_join_as_possible_1_true' > > > > > > > (1910718) panicked at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /var/folders/wq/mxmvfjtd1kjcrm3bdfrn2zd80000gn/T/datafusion-53.0.0.XXXXX.5PoUgcHv4I/apache-datafusion-53.0.0/test-rustup/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-select-58.0.0/src/coalesce/primitive.rs:61:9: > > > > > > > assertion `left == right` failed > > > > > > > left: 4 > > > > > > > right: 2 > > > > > > > > > > > > > > ---- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > joins::hash_join::exec::tests::join_full_multi_batch::batch_size_4_2::use_perfect_hash_join_as_possible_2_false > > > > > > > stdout ---- > > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'joins::hash_join::exec::tests::join_full_multi_batch::batch_size_4_2::use_perfect_hash_join_as_possible_2_false' > > > > > > > (1910719) panicked at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /var/folders/wq/mxmvfjtd1kjcrm3bdfrn2zd80000gn/T/datafusion-53.0.0.XXXXX.5PoUgcHv4I/apache-datafusion-53.0.0/test-rustup/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-select-58.0.0/src/coalesce/primitive.rs:61:9: > > > > > > > assertion `left == right` failed > > > > > > > left: 4 > > > > > > > right: 2 > > > > > > > ``` > > > > > > > > > > > > > > Best, > > > > > > > Kevin Liu > > > > > > > > > > > > > > On Mon, Mar 16, 2026 at 3:52 PM <[email protected]> wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > I would like to propose a release of Apache DataFusion version > > > > > 53.0.0. > > > > > > > > This release candidate is based on commit: > > > > > > > > 28fc91a2a5a8ede5797fde04527f76f351c3cd8b [1] > > > > > > > > The proposed release tarball and signatures are hosted at [2]. > > > > > > > > The changelog is located at [3]. > > > > > > > > Please download, verify checksums and signatures, run the unit > > > > tests, > > > > > > and > > > > > > > > vote > > > > > > > > on the release. The vote will be open for at least 72 hours. > > > > > > > > Only votes from PMC members are binding, but all members of the > > > > > > community > > > > > > > > are > > > > > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > The standard verification procedure is documented at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates > > > > > > > > . > > > > > > > > [ ] +1 Release this as Apache DataFusion 53.0.0 > > > > > > > > [ ] +0 > > > > > > > > [ ] -1 Do not release this as Apache DataFusion 53.0.0 > > because... > > > > > > > > Here is my vote: > > > > > > > > +1 > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion/tree/28fc91a2a5a8ede5797fde04527f76f351c3cd8b > > > > > > > > [2]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-53.0.0-rc2 > > > > > > > > [3]: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion/blob/28fc91a2a5a8ede5797fde04527f76f351c3cd8b/CHANGELOG.md > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
