Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb closed issue #15072: Release DataFusion `47.0.0` (April 2025) URL: https://github.com/apache/datafusion/issues/15072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2835444665 DataFusion 47 is on crates.io https://crates.io/crates/datafusion/47.0.0 So closing this one down -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2816650357 I filed the following ticket for the next release: - https://github.com/apache/datafusion/issues/15771 -- This is an automated message from the Apache Git Service. To respond

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2812715785 Here is a draft upgrade guide: - https://github.com/apache/datafusion/pull/15749 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2810656117 Note that I will be away starting April 18, and so likely can not complete the vote / release process until April 26. @andygrove would it be possible for you to complete the voti

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2810664350 > Note that I will be away starting April 18, and so likely can not complete the vote / release process until April 26. @andygrove would it be possible for you to complete th

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2810649589 I have made a release candidate and started a voting thread: https://lists.apache.org/thread/zrq9x9gf51r8b6m9qokf2q75kh251rm6 -- This is an automated message from the Apache Git

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2810161688 I just merged the version + changelog PR from @xudong963 - https://github.com/apache/datafusion/pull/15731 I also created a `branch-47` here for the release: - https

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2807027482 > @alamb I am hoping that we can merge https://github.com/apache/datafusion/pull/15537 for this release. It was just rebased now that the arrow-rs upgrade is merged. > Tha

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2804744213 > I don't know of anything else we are now waiting on for this release. I suggest we make the release notes PR and generate a release candidate It seems there are still

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

gabotechs commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2804297502 Thanks for putting this together! If we could additionally get https://github.com/apache/datafusion/pull/14412 in, that would be awesome 🙏 -- This is an automated message f

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2802283042 @alamb I am hoping that we can merge https://github.com/apache/datafusion/pull/15537 for this release. It was just rebased now that the arrow-rs upgrade is merged. -- This

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2801302423 I also filed https://github.com/apache/datafusion/issues/15707 to track writing the upgrade guide -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2801294273 Ok, I just merged https://github.com/apache/datafusion/pull/15466 / upgrade to dependencies (arrow/object_store/parquet) 47.0.0 I don't know of anything else we are now wait

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2799902720 Thanks @jayzhan211 for the approval and for the discussion. I'll plan to merge https://github.com/apache/datafusion/pull/15466 tomorrow then unless we want to discuss it further.

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2799561705 I am also +1 for upgrading the dependencies (for selfish reasons; we are waiting on an arrow feature to help with INT96 timestamps in Parquet) -- This is an automated messag

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

jayzhan211 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2799545774 > My only remaining question is if we want to upgrade arrow in this release as well +1 for upgrading all the dependencies -- This is an automated message from the Ap

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2798878082 > > Do we want to hold DF 47 release for the arrow upgrade too? > > I think it is possible (arrow will hopefully be released at the end of this week -- and we could make the DF

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

timsaucer commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2798821873 > > I did a little investigating, but I don't have time for a couple of days to dive in deeper. This appears to be related to [#15542](https://github.com/apache/datafusion/pul

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2798810667 > I did a little investigating, but I don't have time for a couple of days to dive in deeper. This appears to be related to [#15542](https://github.com/apache/datafusion/pull/1554

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

timsaucer commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2798806510 I did a little investigating, but I don't have time for a couple of days to dive in deeper. This appears to be related to https://github.com/apache/datafusion/pull/15542 @UBar

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

timsaucer commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2798792530 I spoke too soon - I'm getting one error in our unit tests on `last_value`. I'm trying to investigate this morning. -- This is an automated message from the Apache Git Servi

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

timsaucer commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2798046180 Running CI on it now: https://github.com/apache/datafusion-python/pull/1104 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

timsaucer commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2797943836 > FYI [@timsaucer](https://github.com/timsaucer) we are getting ready to release datafusion 47 -- shall we test with datafusion-python before doing so? I've been using a

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2797918175 FYI @timsaucer we are getting ready to release datafusion 47 -- shall we test with datafusion-python before doing so? -- This is an automated message from the Apache Git Se

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2797919224 I also tested the upgrade in delta.rs and it seems to have gone well for me - https://github.com/delta-io/delta-rs/pull/3378 -- This is an automated message from the Apache Gi

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2797691953 The upgrade to arrow 55 is now ready for review too: - https://github.com/apache/datafusion/pull/15466 -- This is an automated message from the Apache Git Service. To respond

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2797397320 Thanks -- I plan to make a test PR for delta.rs later this afternoon and will report back -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

Blizzara commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2796559655 > I tested the latest DF against our tests - the Substrait consumer is broken when it comes to renaming Struct fields' insides, due to [#15239 (comment)](https://github.com/apa

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2795994780 To ensure we can release before April 21, I think we can start the vote process in the middle of next week, and after three days, we can release at the end of next week.

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2795986148 > #15676 I added it to the summary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2785553594 Thank you @XiangpengHao , I added the breaking changes that you mentioned to the summary of the issue. -- This is an automated message from the Apache Git Service. To respon

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2794911032 I found a regression in `last_value` behavior that affects Comet: https://github.com/apache/datafusion/issues/15676 -- This is an automated message from the Apache Git Servi

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

XiangpengHao commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2785154065 I have tested the [Parquet viewer](https://parquet-viewer.xiangpeng.systems) with the latest main and found no problems. But I hit a TPC-H panic when running on Liqui

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2784707600 We have started testing Comet with the latest DF from main. I added a link to the Comet in this PR's description. https://github.com/apache/datafusion-comet/pull/1563

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2787171738 > Do we want to hold DF 47 release for the arrow upgrade too? > > I think it is possible (arrow will hopefully be released at the end of this week -- and we could make t

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2787136317 > > FYI, [@andygrove](https://github.com/andygrove) [@alamb](https://github.com/alamb) is working on this: [#15466 (review)](https://github.com/apache/datafusion/pull/15466#pullre

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

adriangb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2786996137 Thanks for pointing that out. Interesting. I'm having trouble thinking of what change between 46 and now would cause that but I'm not surprised in general. Do you have an MRE?

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

XiangpengHao commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2786988859 > I have tested the [Parquet viewer](https://parquet-viewer.xiangpeng.systems) with the latest main and found no problems. > > But I hit a TPC-H panic when running o

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2786958261 > FYI, [@andygrove](https://github.com/andygrove) [@alamb](https://github.com/alamb) is working on this: [#15466 (review)](https://github.com/apache/datafusion/pull/15466#pull

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2786748653 FYI, @andygrove @alamb is working on this: https://github.com/apache/datafusion/pull/15466#pullrequestreview-2750286839 -- This is an automated message from the Apache Git S

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

andygrove commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2786322993 Will we upgrade to Arrow 55 for this release? For the Comet project, we were hoping to get the improved INT96 support so that we can switch over to using DataFusion's ParquetE

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

linhr commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2786105334 I'm working with @shehabgamin to test the latest main branch against Sail. The work is tracked here: So far I've tested the commit

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

Blizzara commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2785876196 I tested the latest DF against our tests - the Substrait consumer is broken when it comes to renaming Struct fields' insides, due to https://github.com/apache/datafusion/pull/1

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2782950272 Makes sense. Thanks @xudong963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2781924081 Hey guys, happy new week, let's start testing the incoming DF47 this week! 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2762088545 > [@alamb](https://github.com/alamb) I think we can start testing the 47.0.0 in the second week of April and begin the release process at the end of that week. What do you think?

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2775012298 > Would really appreciate if could add the following PR to the release as well: > > * [fix: update group by columns for merge phase after spill  #15531](https://github.c

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

rluvaton commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2773417116 Would really appreciate if could add the following PR to the release as well: - #15531 -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2763074215 > For your planning purposes I will be away the week of April 21 -- so perhaps we can start testing a week earlier (week of April 7 so we have time to complete / fix issues pr

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

shehabgamin commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2762517221 Happy to test whenever! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2760208122 @alamb I think we can start testing the 47.0.0 in the second week of April and begin the release process at the end of that week. What do you think? -- This is an automated

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2753175396 The PR https://github.com/apache/datafusion/pull/15266 has significantly improved performance, so I added it to the blog section. -- This is an automated message from the Ap

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2737454899 I believe the voting deadline has passed and I will now will prmote / publish it. THanks again @xudong963 -- This is an automated message from the Apache Git Service. To

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2736259634 > I feel like this may be important enough to try to get into the release. Does anyone else have thoughts? > > * [Comparison Operators for Decimals of Different Precisions a

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

shehabgamin commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2735261314 I feel like this may be important enough to try to get into the release. Does anyone else have thoughts? https://github.com/apache/datafusion/issues/15174 -- This i

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2717257842 > [@XiangpengHao](https://github.com/XiangpengHao) also offered to test with the [parquet viewer](https://parquet-viewer.xiangpeng.systems/) prior to 47: [#15102 (comment)](h

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2713790553 @XiangpengHao also offered to test with the [parquet viewer](https://parquet-viewer.xiangpeng.systems/) prior to 47: https://github.com/apache/datafusion/pull/15102#issuecomment-

Re: [I] Release DataFusion `47.0.0` (April 2025) [datafusion]

xudong963 commented on issue #15072: URL: https://github.com/apache/datafusion/issues/15072#issuecomment-2713344196 @alamb, I'll also be in charge of this release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[I] Release DataFusion `47.0.0` (April 2025) [datafusion]

alamb opened a new issue, #15072: URL: https://github.com/apache/datafusion/issues/15072 ### Is your feature request related to a problem or challenge? ### Is your feature request related to a problem or challenge? Tracking ticket for next release, also a place to track desired