Re: [VOTE] Release Apache DataFusion 45.0.0 RC1
+1 (non binding) Verified on M1 mac running `./dev/release/verify-release-candidate.sh 45.0.0 1` Best, Kevin Liu On Mon, Feb 3, 2025 at 2:00 PM Andrew Lamb wrote: > Hi, > > The releases are coming fast and furious. > > I would like to propose a release of Apache DataFusion version 45.0.0. > > This release candidate is based on commit: > 26058ac024095ad8852eb3a8ab707ac09a02e8d7 [1] > The proposed release tarball and signatures are hosted at [2]. > The changelog is located at [3]. > > Please download, verify checksums and signatures, run the unit tests, and > vote > on the release. The vote will be open for at least 72 hours. > > Only votes from PMC members are binding, but all members of the community > are > encouraged to test the release and vote with "(non-binding)". > > The standard verification procedure is documented at > > https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates > . > > [ ] +1 Release this as Apache DataFusion 45.0.0 > [ ] +0 > [ ] -1 Do not release this as Apache DataFusion 45.0.0 because... > > Here is my vote: > > +1 > > [1]: > > https://github.com/apache/datafusion/tree/26058ac024095ad8852eb3a8ab707ac09a02e8d7 > [2]: > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-45.0.0-rc1 > [3]: > > https://github.com/apache/datafusion/blob/26058ac024095ad8852eb3a8ab707ac09a02e8d7/CHANGELOG.md >
Re: [VOTE] Release DataFusion Python Bindings 44.0.0 RC1
+1 non-binding Ran `./dev/release/verify-release-candidate.sh 44.0.0 1` with the same change to the script as described above. PS, first time running RC verification for datafusion-python, looks like these "Verifying a Release" <https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-a-release> instructions are outdated. Best, Kevin Liu On Mon, Feb 3, 2025 at 9:36 AM L. C. Hsieh wrote: > +1 (binding) > > Verified on M4 Mac. > > Note that after removing requirements-310.txt as Andrew suggested, in > order to make it work, > I also need to add `python3 -m pip install -U maturin` before the > `maturin develop` command in the script. > I left a comment in the PR. > > Thanks Tim and Andrew. > > On Mon, Feb 3, 2025 at 6:15 AM Andrew Lamb wrote: > > > > +1 (binding) > > > > Note I had to update verify_release_candidate.sh manually[1] to get the > > verification to pass. > > > > Also to anyone else interested, Tim and I had a short video call to > verify > > that this is indeed the Tim Saucer I know and thus I have added the key > to > > the KEYS file > > > > However, the key isn't verified by another trusted signature and results > in > > the warning below. This is actually the same for Andy Grove's key as > well, > > so I think we should have a "keysigning party" and add your keys to the > web > > of trust. I will follow up with an email separately > > > > + gpg --verify ./apache-datafusion-python-44.0.0.tar.gz.asc > > ./apache-datafusion-python-44.0.0.tar.gz > > gpg: Signature made Sun Feb 2 17:27:59 2025 EST > > gpg:using EDDSA key > CF6296EE7380F05894FE36443562A212282A90AD > > gpg: Good signature from "Timothy Saucer " > [unknown] > > gpg: WARNING: This key is not certified with a trusted signature! > > gpg: There is no indication that the signature belongs to the > > owner. > > > > > > After fixing that and commenting out the `pip install` (see [1]) the > > verification passed > > > > + popd > > > /var/folders/1l/tg68jc6550gg8xqf1hr4mlwrgn/T/datafusion-python-44.0.0.X.q8U6Obzdcw > > + TEST_SUCCESS=yes > > + echo 'Release candidate looks good!' > > Release candidate looks good! > > + exit 0 > > + cleanup > > + '[' yes = yes ']' > > + rm -fr > > > /var/folders/1l/tg68jc6550gg8xqf1hr4mlwrgn/T/datafusion-python-44.0.0.X.q8U6Obzdcw > > > > > > [1]: https://github.com/apache/datafusion-python/pull/1012 > > > > On Sun, Feb 2, 2025 at 8:05 PM Tim Saucer wrote: > > > > > Ah, I didn’t see that I needed to do that. I’ll upload in the morning. > > > Thank you for catching the issue. > > > > > > > On Feb 2, 2025, at 7:24 PM, Andrew Lamb > wrote: > > > > > > > > Thanks for doing this Tim! > > > > > > > > As we have met in person, perhaps we could/should have a keysigning > party > > > > (aka zoom call) where I can sign and add your keys > > > > > > > > One thing I noticed is that I can't find the key used for for > keysigning > > > > > > > > andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ gpg --verify > > > > apache-datafusion-python-44.0.0.tar.gz.asc > > > > gpg: assuming signed data in 'apache-datafusion-python-44.0.0.tar.gz' > > > > gpg: Signature made Sun Feb 2 17:27:59 2025 EST > > > > gpg:using EDDSA key > > > CF6296EE7380F05894FE36443562A212282A90AD > > > > gpg: Can't check signature: No public key > > > > > > > > I also check on the ubuntu keyserver and it wasn't there either: > > > > > > > > https://keyserver.ubuntu.com/pks/lookup?search=CF6296EE7380F05894FE36443562A212282A90AD&fingerprint=on&op=index > > > > > > > > > > > > This also caused the release verification script for me to fail: > > > > > > > > $ apache-datafusion-python-44.0.0$ > > > > ./dev/release/verify-release-candidate.sh 44.0.0 1 > > > > ... > > > > > > > > + artifact=./apache-datafusion-python-44.0.0.tar.gz > > > > + gpg --verify ./apache-datafusion-python-44.0.0.tar.gz.asc > > > > ./apache-datafusion-python-44.0.0.tar.gz > > > > gpg: Signature made Sun Feb 2 17:27:59 2025 EST > > > > gpg:using EDDSA key > > > CF6296EE7380F05894FE36443562A212282A90AD > > > >
Re: [VOTE] Release Apache DataFusion 45.0.0 RC1
Another data point: I've tested 45.0.0 RC1 against iceberg-rust https://github.com/apache/iceberg-rust/pull/943 Everything passes in CI except one specific doc test. I'm not sure why it fails but I was able to run it successfully on my local machine. Best, Kevin Liu On Mon, Feb 3, 2025 at 4:54 PM Andy Grove wrote: > +1 (binding) > > Verified on Mac. Also verified with Comet CI in > https://github.com/apache/datafusion-comet/pull/1364 > > Thanks, Andrew! > > On Mon, Feb 3, 2025 at 3:45 PM L. C. Hsieh wrote: > > > +1 (binding) > > > > Verified on M4 Mac. > > > > Thanks Andrew. > > > > On Mon, Feb 3, 2025 at 1:59 PM Andrew Lamb wrote: > > > > > > Hi, > > > > > > The releases are coming fast and furious. > > > > > > I would like to propose a release of Apache DataFusion version 45.0.0. > > > > > > This release candidate is based on commit: > > > 26058ac024095ad8852eb3a8ab707ac09a02e8d7 [1] > > > The proposed release tarball and signatures are hosted at [2]. > > > The changelog is located at [3]. > > > > > > Please download, verify checksums and signatures, run the unit tests, > and > > > vote > > > on the release. The vote will be open for at least 72 hours. > > > > > > Only votes from PMC members are binding, but all members of the > community > > > are > > > encouraged to test the release and vote with "(non-binding)". > > > > > > The standard verification procedure is documented at > > > > > > https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates > > > . > > > > > > [ ] +1 Release this as Apache DataFusion 45.0.0 > > > [ ] +0 > > > [ ] -1 Do not release this as Apache DataFusion 45.0.0 because... > > > > > > Here is my vote: > > > > > > +1 > > > > > > [1]: > > > > > > https://github.com/apache/datafusion/tree/26058ac024095ad8852eb3a8ab707ac09a02e8d7 > > > [2]: > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-45.0.0-rc1 > > > [3]: > > > > > > https://github.com/apache/datafusion/blob/26058ac024095ad8852eb3a8ab707ac09a02e8d7/CHANGELOG.md > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > > For additional commands, e-mail: dev-h...@datafusion.apache.org > > > > >
Re: [RESULT][VOTE] Release DataFusion Python Bindings 44.0.0 RC1
Late to the party, I just joined the devlist. Thanks everyone 🎉! I see v44.0.0 in pypi https://pypi.org/project/datafusion/ Best, Kevin Liu On Fri, Feb 7, 2025 at 7:42 AM Andrew Lamb wrote: > 🎉 > > On Fri, Feb 7, 2025 at 10:40 AM Tim Saucer wrote: > > > Thank you, Andrew. Andy gave me pypi access and I've published it. > > > > Best, > > Tim > > > > On Fri, Feb 7, 2025 at 9:01 AM Andrew Lamb wrote: > > > > > With 5+1 votes (3 binding) the release is approved! > > > > > > Success! The release is available here: > > > > > > > > > > > > https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-44.0.0 > > > > > > I did not upload the artifacts to pypi / other repositories as I don't > > know > > > how to do so. Maybe Tim you can help here. > > > > > > Thank you so much for everyone who helped with this release > > > > > > Andrew > > > > > > On Thu, Feb 6, 2025 at 7:40 PM Andy Grove > wrote: > > > > > > > +1 (binding) > > > > > > > > Verified on Mac. > > > > > > > > Thanks, Tim! > > > > > > > > On Sun, Feb 2, 2025 at 3:15 PM Tim Saucer > wrote: > > > > > > > > > Hi, > > > > > > > > > > I would like to propose a release of the Apache DataFusion Python > > > > Bindings, > > > > > version 44.0.0. > > > > > > > > > > This release candidate is based on commit: > > > > > f5a9f25080a6774d9dd202774baae8a659b2e396 [1] > > > > > The proposed release tarball and signatures are hosted at [2]. > > > > > The changelog is located at [3]. > > > > > The Python wheels are located at [4]. > > > > > > > > > > Please download, verify checksums and signatures, run the unit > tests, > > > and > > > > > vote > > > > > on the release. The vote will be open for at least 72 hours. > > > > > > > > > > Only votes from PMC members are binding, but all members of the > > > community > > > > > are > > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > > > The standard verification procedure is documented at > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates > > > > > . > > > > > > > > > > [ ] +1 Release this as Apache DataFusion Python 44.0.0 > > > > > [ ] +0 > > > > > [ ] -1 Do not release this as Apache DataFusion Python 44.0.0 > > > because... > > > > > > > > > > Here is my vote: > > > > > > > > > > +1 (non-binding) > > > > > > > > > > [1]: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion-python/tree/f5a9f25080a6774d9dd202774baae8a659b2e396 > > > > > [2]: > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-44.0.0-rc1 > > > > > [3]: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion-python/blob/f5a9f25080a6774d9dd202774baae8a659b2e396/CHANGELOG.md > > > > > [4]: https://test.pypi.org/project/datafusion/44.0.0/ > > > > > > > > > > > > > > >
Re: [RESULT] [VOTE] Release Apache DataFusion 45.0.0 RC1
Cheers! Thanks for all the hard work and the super fast release process. Best, Kevin Liu On Fri, Feb 7, 2025 at 7:50 AM Andrew Lamb wrote: > (resending with correct subject) > > On Fri, Feb 7, 2025 at 10:49 AM Andrew Lamb wrote: > > > With 4 +1 votes (3 binding) the release is approved! > > > > The release is available here: > > > https://dist.apache.org/repos/dist/release/datafusion/datafusion-45.0.0 > > > > I have also published the release on crates.io: > > https://crates.io/crates/datafusion/45.0.0 > > > > Thanks everyone for your help / support -- I hope this one is epic (and a > > low drama release) > > > > Andrew > > > > On Tue, Feb 4, 2025 at 12:37 PM Andrew Lamb > wrote: > > > >> Awesome -- thank you for the report > >> > >> On Tue, Feb 4, 2025 at 12:19 PM Kevin Liu > wrote: > >> > >>> Another data point: > >>> I've tested 45.0.0 RC1 against iceberg-rust > >>> https://github.com/apache/iceberg-rust/pull/943 > >>> > >>> Everything passes in CI except one specific doc test. I'm not sure why > it > >>> fails but I was able to run it successfully on my local machine. > >>> > >>> Best, > >>> Kevin Liu > >>> > >>> On Mon, Feb 3, 2025 at 4:54 PM Andy Grove > wrote: > >>> > >>> > +1 (binding) > >>> > > >>> > Verified on Mac. Also verified with Comet CI in > >>> > https://github.com/apache/datafusion-comet/pull/1364 > >>> > > >>> > Thanks, Andrew! > >>> > > >>> > On Mon, Feb 3, 2025 at 3:45 PM L. C. Hsieh wrote: > >>> > > >>> > > +1 (binding) > >>> > > > >>> > > Verified on M4 Mac. > >>> > > > >>> > > Thanks Andrew. > >>> > > > >>> > > On Mon, Feb 3, 2025 at 1:59 PM Andrew Lamb > >>> wrote: > >>> > > > > >>> > > > Hi, > >>> > > > > >>> > > > The releases are coming fast and furious. > >>> > > > > >>> > > > I would like to propose a release of Apache DataFusion version > >>> 45.0.0. > >>> > > > > >>> > > > This release candidate is based on commit: > >>> > > > 26058ac024095ad8852eb3a8ab707ac09a02e8d7 [1] > >>> > > > The proposed release tarball and signatures are hosted at [2]. > >>> > > > The changelog is located at [3]. > >>> > > > > >>> > > > Please download, verify checksums and signatures, run the unit > >>> tests, > >>> > and > >>> > > > vote > >>> > > > on the release. The vote will be open for at least 72 hours. > >>> > > > > >>> > > > Only votes from PMC members are binding, but all members of the > >>> > community > >>> > > > are > >>> > > > encouraged to test the release and vote with "(non-binding)". > >>> > > > > >>> > > > The standard verification procedure is documented at > >>> > > > > >>> > > > >>> > > >>> > https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates > >>> > > > . > >>> > > > > >>> > > > [ ] +1 Release this as Apache DataFusion 45.0.0 > >>> > > > [ ] +0 > >>> > > > [ ] -1 Do not release this as Apache DataFusion 45.0.0 because... > >>> > > > > >>> > > > Here is my vote: > >>> > > > > >>> > > > +1 > >>> > > > > >>> > > > [1]: > >>> > > > > >>> > > > >>> > > >>> > https://github.com/apache/datafusion/tree/26058ac024095ad8852eb3a8ab707ac09a02e8d7 > >>> > > > [2]: > >>> > > > > >>> > > > >>> > > >>> > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-45.0.0-rc1 > >>> > > > [3]: > >>> > > > > >>> > > > >>> > > >>> > https://github.com/apache/datafusion/blob/26058ac024095ad8852eb3a8ab707ac09a02e8d7/CHANGELOG.md > >>> > > > >>> > > > - > >>> > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > >>> > > For additional commands, e-mail: dev-h...@datafusion.apache.org > >>> > > > >>> > > > >>> > > >>> > >> >
Re: [RESULT][VOTE] Release DataFusion Python Bindings 44.0.0 RC1
Hey folks, Quick question on this release, should the datafusion-python crate be updated as well? I see the latest version is currently v43.1.0 https://crates.io/crates/datafusion-python Looking back at the devlist, I didn't see a thread specifically for releasing the crate. Best, Kevin Liu On Fri, Feb 7, 2025 at 7:51 AM Kevin Liu wrote: > Late to the party, I just joined the devlist. > Thanks everyone 🎉! I see v44.0.0 in pypi > https://pypi.org/project/datafusion/ > > Best, > Kevin Liu > > On Fri, Feb 7, 2025 at 7:42 AM Andrew Lamb wrote: > >> 🎉 >> >> On Fri, Feb 7, 2025 at 10:40 AM Tim Saucer wrote: >> >> > Thank you, Andrew. Andy gave me pypi access and I've published it. >> > >> > Best, >> > Tim >> > >> > On Fri, Feb 7, 2025 at 9:01 AM Andrew Lamb >> wrote: >> > >> > > With 5+1 votes (3 binding) the release is approved! >> > > >> > > Success! The release is available here: >> > > >> > > >> > > >> > >> https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-44.0.0 >> > > >> > > I did not upload the artifacts to pypi / other repositories as I don't >> > know >> > > how to do so. Maybe Tim you can help here. >> > > >> > > Thank you so much for everyone who helped with this release >> > > >> > > Andrew >> > > >> > > On Thu, Feb 6, 2025 at 7:40 PM Andy Grove >> wrote: >> > > >> > > > +1 (binding) >> > > > >> > > > Verified on Mac. >> > > > >> > > > Thanks, Tim! >> > > > >> > > > On Sun, Feb 2, 2025 at 3:15 PM Tim Saucer >> wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > I would like to propose a release of the Apache DataFusion Python >> > > > Bindings, >> > > > > version 44.0.0. >> > > > > >> > > > > This release candidate is based on commit: >> > > > > f5a9f25080a6774d9dd202774baae8a659b2e396 [1] >> > > > > The proposed release tarball and signatures are hosted at [2]. >> > > > > The changelog is located at [3]. >> > > > > The Python wheels are located at [4]. >> > > > > >> > > > > Please download, verify checksums and signatures, run the unit >> tests, >> > > and >> > > > > vote >> > > > > on the release. The vote will be open for at least 72 hours. >> > > > > >> > > > > Only votes from PMC members are binding, but all members of the >> > > community >> > > > > are >> > > > > encouraged to test the release and vote with "(non-binding)". >> > > > > >> > > > > The standard verification procedure is documented at >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates >> > > > > . >> > > > > >> > > > > [ ] +1 Release this as Apache DataFusion Python 44.0.0 >> > > > > [ ] +0 >> > > > > [ ] -1 Do not release this as Apache DataFusion Python 44.0.0 >> > > because... >> > > > > >> > > > > Here is my vote: >> > > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > [1]: >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/datafusion-python/tree/f5a9f25080a6774d9dd202774baae8a659b2e396 >> > > > > [2]: >> > > > > >> > > > > >> > > > >> > > >> > >> https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-44.0.0-rc1 >> > > > > [3]: >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/datafusion-python/blob/f5a9f25080a6774d9dd202774baae8a659b2e396/CHANGELOG.md >> > > > > [4]: https://test.pypi.org/project/datafusion/44.0.0/ >> > > > > >> > > > >> > > >> > >> >
Re: [DISCUSS] Blog on Parquet Pruning
Thanks for posting here! Great read. Best, Kevin Liu On Tue, Mar 18, 2025 at 1:46 PM Andrew Lamb wrote: > There is a PR to cross post a blog about Parquet pruning on the DataFusion > site[1]. Please take a look and leave comments if you are interested. > > [1]: https://github.com/apache/datafusion-site/pull/60 >
Re: [ANNOUNCE] New DataFusion committer: Marko Milenković (milenkovicm)
Congratulations! Great to see the community growing. Best, Kevin Liu On Tue, Mar 18, 2025 at 2:05 AM Tim Saucer wrote: > Congratulations! So well deserved! > > Best, > Tim > > On Mon, Mar 17, 2025 at 8:22 PM Andrew Lamb > wrote: > > > On behalf of the DataFusion PMC, I'm happy to announce that Marko > > Milenković > > has accepted an invitation to become a committer on Apache > > DataFusion. Welcome, and thank you for your contributions! > > > > The PMC > > >
Re: [VOTE] Release DataFusion Python Bindings 46.0.0 RC1
+1 (non-binding) Thanks, Kevin Liu On Sun, Mar 23, 2025 at 8:39 PM Xudong Wang wrote: > +1 (non-binding) > > Verified on my M4 Mac > > Thanks, Tim. > > On 2025/03/23 00:35:43 Tim Saucer wrote: > > Hi, > > > > I would like to propose a release of the Apache DataFusion Python > Bindings, > > version 46.0.0. > > > > This release candidate is based on commit: > > 1aa9cb98bc2cc070048027ac96658c69958db516 [1] > > The proposed release tarball and signatures are hosted at [2]. > > The changelog is located at [3]. > > The Python wheels are located at [4]. > > > > Please download, verify checksums and signatures, run the unit tests, and > > vote > > on the release. The vote will be open for at least 72 hours. > > > > Only votes from PMC members are binding, but all members of the community > > are > > encouraged to test the release and vote with "(non-binding)". > > > > The standard verification procedure is documented at > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates > > . > > > > [ ] +1 Release this as Apache DataFusion Python 46.0.0 > > [ ] +0 > > [ ] -1 Do not release this as Apache DataFusion Python 46.0.0 because... > > > > Here is my vote: > > > > +1 > > > > [1]: > > > https://github.com/apache/datafusion-python/tree/1aa9cb98bc2cc070048027ac96658c69958db516 > > [2]: > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-46.0.0-rc1 > > [3]: > > > https://github.com/apache/datafusion-python/blob/1aa9cb98bc2cc070048027ac96658c69958db516/CHANGELOG.md > > [4]: https://test.pypi.org/project/datafusion/46.0.0/ > > > > - > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > For additional commands, e-mail: dev-h...@datafusion.apache.org > >
Re: joining the mailing list
Hi Sami, Here's the instruction to join the mailing list https://datafusion.apache.org/contributor-guide/communication.html#mailing-lists You'd have to send an email to `dev-subscr...@datafusion.apache.org` and then follow the instructions. Best, Kevin Liu On Thu, Mar 27, 2025 at 1:35 PM Sami Hadouaj wrote: > Hi, > > I would like to join this mailing list. > > -- > Sami Hadouaj > PhD Candidate > Computer and Information Science > University of Michigan - Rackham Graduate School >
Re: [VOTE] Release DataFusion Python Bindings 45.2.0 RC2
Hey folks, Following up on the conversation about pyarrow, v19.0.1 has just been released on pypi, https://pypi.org/project/pyarrow/. I reran the test suite using `pyarrow==19.0.1` and verified that all tests passed. Best, Kevin Liu On Tue, Feb 18, 2025 at 3:04 AM Andrew Lamb wrote: > +1 (binding) > > I ran the release verification script and it worked well for me on a M3 mac > > Thanks Tim and everyone else who is verifying these releases (Kevin it is > great to see you here, and L.C. your voting participation is amazing!) > > Andrew > > On Mon, Feb 17, 2025 at 8:01 PM Kevin Liu wrote: > > > +1 (non-binding) Thanks for running this release! > > > > I was able to run the verification script successfully on M1 Mac. > > ``` > > ./dev/release/verify-release-candidate.sh 45.2.0 2 > > ``` > > > > I also verified by running pytest directly, based on > > > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-a-release > > , > > after downgrading to `pyarrow==18.0.0` > > ``` > > git fetch --tags > > git checkout 45.2.0-rc2 > > git submodule update --init --recursive > > python3 -m venv .venv > > source .venv/bin/activate > > pip install --extra-index-url https://test.pypi.org/simple/ > > datafusion==45.2.0 > > pip install pytest > > pip install pyarrow==18.0.0 > > pytest --import-mode=importlib python/tests > > ``` > > > > For context, here's the error I encountered using `pyarrow=19.0.0`, > > https://gist.github.com/kevinjqliu/6d111bf7bd74c165c34e4366fe53a6c6. > > Looks like we don't pin a specific pyarrow version, so when the new > version > > (`19.0.1`) is released next week, the issue should be automatically > > resolved. > > > > Best, > > Kevin Liu > > > > On Mon, Feb 17, 2025 at 3:32 PM L. C. Hsieh wrote: > > > > > +1 (binding) > > > > > > Verified on M4 Mac. > > > > > > Thanks Tim. > > > > > > On Mon, Feb 17, 2025 at 3:15 PM Tim Saucer > wrote: > > > > > > > > Hi, > > > > > > > > I would like to propose a release of the Apache DataFusion Python > > > Bindings, > > > > version 45.2.0. > > > > > > > > I have two special notes for this release > > > > - The version is 45.2.0 due to changes I needed to make during > release > > > > candidate testing prior to submitting this for a vote. I would like > to > > > > revisit next release how we compose release candidates to prevent > these > > > > minor version numbers from diverging too far from the upstream > > DataFusion > > > > repository. > > > > - There is currently a bug in pyarrow-19.0.0. This is a known issue > > > > described at [5] that will be updated very soon. When running the > > > > validation you will need to do an additional step of `pip install > > > > pyarrow==18.0.0` to downgrade pyarrow. > > > > > > > > This release candidate is based on commit: > > > > 639e2abc9ce1e7753f9779c7496da799a32c54a5 [1] > > > > The proposed release tarball and signatures are hosted at [2]. > > > > The changelog is located at [3]. > > > > The Python wheels are located at [4]. > > > > > > > > Please download, verify checksums and signatures, run the unit tests, > > and > > > > vote > > > > on the release. The vote will be open for at least 72 hours. > > > > > > > > Only votes from PMC members are binding, but all members of the > > community > > > > are > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > The standard verification procedure is documented at > > > > > > > > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates > > > > . > > > > > > > > [ ] +1 Release this as Apache DataFusion Python 45.2.0 > > > > [ ] +0 > > > > [ ] -1 Do not release this as Apache DataFusion Python 45.2.0 > > because... > > > > > > > > Here is my vote: > > > > > > > > +1 (non-binding) > > > > > > > > [1]: > > > > > > > > > > https://github.com/apache/datafusion-python/tree/639e2abc9ce1e7753f9779c7496da799a32c54a5 > > > > [2]: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-45.2.0-rc2 > > > > [3]: > > > > > > > > > > https://github.com/apache/datafusion-python/blob/639e2abc9ce1e7753f9779c7496da799a32c54a5/CHANGELOG.md > > > > [4]: https://test.pypi.org/project/datafusion/45.2.0/ > > > > [5]: > > > > > > > > > > https://arrow.apache.org/blog/2025/01/16/19.0.0-release/#release-highlights > > > > - > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > > > For additional commands, e-mail: dev-h...@datafusion.apache.org > > > > > > > > >
Re: [VOTE] Release DataFusion Python Bindings 45.2.0 RC2
+1 (non-binding) Thanks for running this release! I was able to run the verification script successfully on M1 Mac. ``` ./dev/release/verify-release-candidate.sh 45.2.0 2 ``` I also verified by running pytest directly, based on https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-a-release, after downgrading to `pyarrow==18.0.0` ``` git fetch --tags git checkout 45.2.0-rc2 git submodule update --init --recursive python3 -m venv .venv source .venv/bin/activate pip install --extra-index-url https://test.pypi.org/simple/ datafusion==45.2.0 pip install pytest pip install pyarrow==18.0.0 pytest --import-mode=importlib python/tests ``` For context, here's the error I encountered using `pyarrow=19.0.0`, https://gist.github.com/kevinjqliu/6d111bf7bd74c165c34e4366fe53a6c6. Looks like we don't pin a specific pyarrow version, so when the new version (`19.0.1`) is released next week, the issue should be automatically resolved. Best, Kevin Liu On Mon, Feb 17, 2025 at 3:32 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M4 Mac. > > Thanks Tim. > > On Mon, Feb 17, 2025 at 3:15 PM Tim Saucer wrote: > > > > Hi, > > > > I would like to propose a release of the Apache DataFusion Python > Bindings, > > version 45.2.0. > > > > I have two special notes for this release > > - The version is 45.2.0 due to changes I needed to make during release > > candidate testing prior to submitting this for a vote. I would like to > > revisit next release how we compose release candidates to prevent these > > minor version numbers from diverging too far from the upstream DataFusion > > repository. > > - There is currently a bug in pyarrow-19.0.0. This is a known issue > > described at [5] that will be updated very soon. When running the > > validation you will need to do an additional step of `pip install > > pyarrow==18.0.0` to downgrade pyarrow. > > > > This release candidate is based on commit: > > 639e2abc9ce1e7753f9779c7496da799a32c54a5 [1] > > The proposed release tarball and signatures are hosted at [2]. > > The changelog is located at [3]. > > The Python wheels are located at [4]. > > > > Please download, verify checksums and signatures, run the unit tests, and > > vote > > on the release. The vote will be open for at least 72 hours. > > > > Only votes from PMC members are binding, but all members of the community > > are > > encouraged to test the release and vote with "(non-binding)". > > > > The standard verification procedure is documented at > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates > > . > > > > [ ] +1 Release this as Apache DataFusion Python 45.2.0 > > [ ] +0 > > [ ] -1 Do not release this as Apache DataFusion Python 45.2.0 because... > > > > Here is my vote: > > > > +1 (non-binding) > > > > [1]: > > > https://github.com/apache/datafusion-python/tree/639e2abc9ce1e7753f9779c7496da799a32c54a5 > > [2]: > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-45.2.0-rc2 > > [3]: > > > https://github.com/apache/datafusion-python/blob/639e2abc9ce1e7753f9779c7496da799a32c54a5/CHANGELOG.md > > [4]: https://test.pypi.org/project/datafusion/45.2.0/ > > [5]: > > > https://arrow.apache.org/blog/2025/01/16/19.0.0-release/#release-highlights > > - > > - > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > For additional commands, e-mail: dev-h...@datafusion.apache.org > >
Re: [RESULT][VOTE] Release DataFusion Python Bindings 44.0.0 RC1
Thank you! Appreciate the quick response here. Best, Kevin Liu On Tue, Feb 11, 2025 at 5:17 PM Andrew Lamb wrote: > I have published version 44 now to crates.io: > https://crates.io/crates/datafusion-python/44.0.0 > > Thanks for the ping Kevin > > Andrew > > On Tue, Feb 11, 2025 at 2:45 PM Andrew Lamb > wrote: > > > I think it is an oversight -- it appears only Andy Grove is the only > owner > > of that crate so only he can publish it. I will see if we can get access > to > > some others to help > > > > On Tue, Feb 11, 2025 at 11:51 AM Kevin Liu > wrote: > > > > > Hey folks, > > > > > > Quick question on this release, should the datafusion-python crate be > > > updated as well? I see the latest version is currently v43.1.0 > > > https://crates.io/crates/datafusion-python > > > Looking back at the devlist, I didn't see a thread specifically for > > > releasing the crate. > > > > > > Best, > > > Kevin Liu > > > > > > On Fri, Feb 7, 2025 at 7:51 AM Kevin Liu > wrote: > > > > > > > Late to the party, I just joined the devlist. > > > > Thanks everyone 🎉! I see v44.0.0 in pypi > > > > https://pypi.org/project/datafusion/ > > > > > > > > Best, > > > > Kevin Liu > > > > > > > > On Fri, Feb 7, 2025 at 7:42 AM Andrew Lamb > > wrote: > > > > > > > >> 🎉 > > > >> > > > >> On Fri, Feb 7, 2025 at 10:40 AM Tim Saucer > > wrote: > > > >> > > > >> > Thank you, Andrew. Andy gave me pypi access and I've published it. > > > >> > > > > >> > Best, > > > >> > Tim > > > >> > > > > >> > On Fri, Feb 7, 2025 at 9:01 AM Andrew Lamb > > > >> wrote: > > > >> > > > > >> > > With 5+1 votes (3 binding) the release is approved! > > > >> > > > > > >> > > Success! The release is available here: > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > > > > https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-44.0.0 > > > >> > > > > > >> > > I did not upload the artifacts to pypi / other repositories as I > > > don't > > > >> > know > > > >> > > how to do so. Maybe Tim you can help here. > > > >> > > > > > >> > > Thank you so much for everyone who helped with this release > > > >> > > > > > >> > > Andrew > > > >> > > > > > >> > > On Thu, Feb 6, 2025 at 7:40 PM Andy Grove < > andygrov...@gmail.com> > > > >> wrote: > > > >> > > > > > >> > > > +1 (binding) > > > >> > > > > > > >> > > > Verified on Mac. > > > >> > > > > > > >> > > > Thanks, Tim! > > > >> > > > > > > >> > > > On Sun, Feb 2, 2025 at 3:15 PM Tim Saucer < > timsau...@gmail.com> > > > >> wrote: > > > >> > > > > > > >> > > > > Hi, > > > >> > > > > > > > >> > > > > I would like to propose a release of the Apache DataFusion > > > Python > > > >> > > > Bindings, > > > >> > > > > version 44.0.0. > > > >> > > > > > > > >> > > > > This release candidate is based on commit: > > > >> > > > > f5a9f25080a6774d9dd202774baae8a659b2e396 [1] > > > >> > > > > The proposed release tarball and signatures are hosted at > [2]. > > > >> > > > > The changelog is located at [3]. > > > >> > > > > The Python wheels are located at [4]. > > > >> > > > > > > > >> > > > > Please download, verify checksums and signatures, run the > unit > > > >> tests, > > > >> > > and > > > >> > > > > vote > > > >> > > > > on the release. The vote will be open for at least 72 hours. > > > >> > > > > > > > >> > > > > Only votes from PMC members are binding, but all members of > > the > > > >> > > community > > > >> > > > > are > > > >> > > > > encouraged to test the release and vote with > "(non-binding)". > > > >> > > > > > > > >> > > > > The standard verification procedure is documented at > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-release-candidates > > > >> > > > > . > > > >> > > > > > > > >> > > > > [ ] +1 Release this as Apache DataFusion Python 44.0.0 > > > >> > > > > [ ] +0 > > > >> > > > > [ ] -1 Do not release this as Apache DataFusion Python > 44.0.0 > > > >> > > because... > > > >> > > > > > > > >> > > > > Here is my vote: > > > >> > > > > > > > >> > > > > +1 (non-binding) > > > >> > > > > > > > >> > > > > [1]: > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/datafusion-python/tree/f5a9f25080a6774d9dd202774baae8a659b2e396 > > > >> > > > > [2]: > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-44.0.0-rc1 > > > >> > > > > [3]: > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/datafusion-python/blob/f5a9f25080a6774d9dd202774baae8a659b2e396/CHANGELOG.md > > > >> > > > > [4]: https://test.pypi.org/project/datafusion/44.0.0/ > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
Re: [VOTE] Release Apache DataFusion Comet 0.6.0 RC1
Hey folks, First time running release verification for datafusion-comet. I'm following the instructions here https://github.com/apache/datafusion-comet/blob/main/dev/release/verifying-release-candidates.md, and running on the latest main branch. The script was not able to download the artifact from `dist.apache.org`. In fact, I don't see anything related to datafusion-comet under ` https://dist.apache.org/repos/dist/dev/datafusion/` Here's the log: ``` ➜ datafusion-comet git:(main) ./dev/release/verify-release-candidate.sh 0.6.0 1 ... + fetch_archive apache-datafusion-comet-0.6.0 + local dist_name=apache-datafusion-comet-0.6.0 + download_rc_file apache-datafusion-comet-0.6.0.tar.gz + download_dist_file apache-datafusion-comet-0.6.0-rc1/apache-datafusion-comet-0.6.0.tar.gz + curl --silent --show-error --fail --location --remote-name https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-comet-0.6.0-rc1/apache-datafusion-comet-0.6.0.tar.gz curl: (22) The requested URL returned error: 404 + cleanup + '[' no = yes ']' + echo 'Failed to verify release candidate. See /var/folders/f1/3_vzsn7x1jq9hszb3z9y6f0mgn/T/datafusion-comet-0.6.0.X.3RlR7ejtd9 for details.' Failed to verify release candidate. See /var/folders/f1/3_vzsn7x1jq9hszb3z9y6f0mgn/T/datafusion-comet-0.6.0.X.3RlR7ejtd9 for details. ``` Please let me know if I'm missing something. Best, Kevin Liu On Fri, Feb 14, 2025 at 12:21 PM L. C. Hsieh wrote: > +1 (binding) > > Verified on M4 Mac. > > Thanks Andy. > > On Fri, Feb 14, 2025 at 11:27 AM Andy Grove wrote: > > > > Hi, > > > > I would like to propose a release of Apache DataFusion Comet version > 0.6.0. > > > > This release candidate is based on commit: > > b735f2debb983e96d8f4a8a948ef4d61f80f6bcb [1] > > The proposed release tarball and signatures are hosted at [2]. > > Pre-built jar files are available in a Maven staging repository [3]. > > The changelog is located at [4]. > > > > Please download, verify checksums and signatures, run the unit tests, and > > vote > > on the release. The vote will be open for at least 72 hours. > > > > Only votes from PMC members are binding, but all members of the community > > are > > encouraged to test the release and vote with "(non-binding)". > > > > The standard verification procedure is documented at > > > https://github.com/apache/datafusion-comet/blob/main/dev/release/verifying-release-candidates.md > > > > [ ] +1 Release this as Apache DataFusion Comet 0.6.0 > > [ ] +0 > > [ ] -1 Do not release this as Apache DataFusion Comet 0.6.0 because... > > > > Here is my vote: > > > > +1 > > > > [1]: > > > https://github.com/apache/datafusion-comet/tree/b735f2debb983e96d8f4a8a948ef4d61f80f6bcb > > [2]: > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-comet-0.6.0-rc1 > > [3]: > https://repository.apache.org/#nexus-search;quick~org.apache.datafusion > > [4]: > > > https://github.com/apache/datafusion-comet/blob/b735f2debb983e96d8f4a8a948ef4d61f80f6bcb/CHANGELOG.md > > - > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > For additional commands, e-mail: dev-h...@datafusion.apache.org > >
Re: [VOTE] Release Apache DataFusion 46.0.0 RC2
Thank you, that fixed the issue.
+1 (non-binding).
Verified by running `./dev/release/verify-release-candidate.sh 46.0.0 2` on
M1 Mac.
Best,
Kevin Liu
On Tue, Mar 4, 2025 at 8:48 AM Andrew Lamb wrote:
> I think you need to set the limit of files higher. Something like the
> following
>
> ```
> ulimit -n 1
> ```
>
> We discussed this on a ticket previously[1]. Maybe we should fix it for
> real :thinking:
>
> [1] https://github.com/apache/datafusion/issues/10674
>
>
>
> On Tue, Mar 4, 2025 at 11:30 AM Kevin Liu wrote:
>
> > Hey folks,
> >
> > I'm having issues with one particular test. It could be my local
> machine. I
> > tried restarting my laptop and running the test in isolation (`cargo test
> > -p datafusion --test fuzz`) but the error still persists.
> >
> > ```
> > ./dev/release/verify-release-candidate.sh 46.0.0 2
> >
> > failures:
> >
> > fuzz_cases::sort_fuzz::test_sort_10k_mem stdout
> >
> > thread 'fuzz_cases::sort_fuzz::test_sort_10k_mem' panicked at
> > datafusion/core/tests/fuzz_cases/sort_fuzz.rs:272:63:
> > called `Result::unwrap()` on an `Err` value: Execution("Failed to create
> > partition file at
> >
> \"/var/folders/f1/3_vzsn7x1jq9hszb3z9y6f0mgn/T/.tmpYjVAvj/.tmph79STL\":
> > Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
> > note: run with `RUST_BACKTRACE=1` environment variable to display a
> > backtrace
> >
> >
> > failures:
> > fuzz_cases::sort_fuzz::test_sort_10k_mem
> >
> > test result: FAILED. 64 passed; 1 failed; 0 ignored; 0 measured; 0
> filtered
> > out; finished in 74.07s
> > ```
> >
> > Best,
> > Kevin Liu
> >
> > On Tue, Mar 4, 2025 at 7:21 AM L. C. Hsieh wrote:
> >
> > > +1 (binding)
> > >
> > > Verified on M4 Mac.
> > >
> > > Thanks Xudong.
> > >
> > > On Tue, Mar 4, 2025 at 7:02 AM Andrew Lamb
> wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > I ran the verification script on an M3 mac. Thank you so much Xudong!
> > > >
> > > > Andrew
> > > >
> > > > On Tue, Mar 4, 2025 at 9:47 AM Wang Xudong
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would like to propose a release of Apache DataFusion version
> > 46.0.0.
> > > > >
> > > > > This release candidate is based on commit:
> > > > > d5ca8307940c1a6345419a2c8d91ef87704659be [1]
> > > > > The proposed release tarball and signatures are hosted at [2].
> > > > > The changelog is located at [3].
> > > > >
> > > > > Please download, verify checksums and signatures, run the unit
> tests,
> > > and
> > > > > vote
> > > > > on the release. The vote will be open for at least 72 hours.
> > > > >
> > > > > Only votes from PMC members are binding, but all members of the
> > > community
> > > > > are
> > > > > encouraged to test the release and vote with "(non-binding)".
> > > > >
> > > > > The standard verification procedure is documented at
> > > > >
> > > > >
> > >
> >
> https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates
> > > > > .
> > > > >
> > > > > [ ] +1 Release this as Apache DataFusion 46.0.0
> > > > > [ ] +0
> > > > > [ ] -1 Do not release this as Apache DataFusion 46.0.0 because...
> > > > >
> > > > > Here is my vote:
> > > > >
> > > > > +1
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > >
> >
> https://github.com/apache/datafusion/tree/d5ca8307940c1a6345419a2c8d91ef87704659be
> > > > > [2]:
> > > > >
> > > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-46.0.0-rc2
> > > > > [3]:
> > > > >
> > > > >
> > >
> >
> https://github.com/apache/datafusion/blob/d5ca8307940c1a6345419a2c8d91ef87704659be/CHANGELOG.md
> > > > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org
> > > For additional commands, e-mail: dev-h...@datafusion.apache.org
> > >
> > >
> >
>
Re: [VOTE] Release Apache DataFusion sqlparser-rs 0.55.0 RC1
+1 non binding. Ran `./dev/release/verify-release-candidate.sh 0.55.0 1` on M1 Mac. Ran an example, `echo "select 1;" | cargo run --example cli -` Best, Kevin Liu On Thu, Feb 27, 2025 at 11:30 AM L. C. Hsieh wrote: > +1 (binding) > > Verified on M4 Mac. > > Thanks Andrew. > > > On Thu, Feb 27, 2025 at 11:08 AM Andrew Lamb wrote: > > > > Hi, > > > > I would like to propose a release of Apache DataFusion sqlparser-rs > version > > 0.55.0. > > > > This release candidate is based on commit: > > ed416548dcfe4a73a3240bbf625fb9010a4925c8 [1] > > The proposed release tarball and signatures are hosted at [2]. > > The changelog is located at [3]. > > > > Please download, verify checksums and signatures, run the unit tests, and > > vote > > on the release. The vote will be open for at least 72 hours. > > > > Only votes from PMC members are binding, but all members of the community > > are > > encouraged to test the release and vote with "(non-binding)". > > > > The standard verification procedure is documented at > > > https://github.com/apache/datafusion-sqlparser-rs/blob/main/dev/release/README.md#verifying-release-candidates > > . > > > > [ ] +1 Release this as Apache DataFusion sqlparser-rs 0.55.0 > > [ ] +0 > > [ ] -1 Do not release this as Apache DataFusion sqlparser-rs 0.55.0 > > because... > > > > Here is my vote: > > > > +1 > > > > [1]: > > > https://github.com/apache/datafusion-sqlparser-rs/tree/ed416548dcfe4a73a3240bbf625fb9010a4925c8 > > [2]: > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-sqlparser-rs-0.55.0-rc1 > > [3]: > > > https://github.com/apache/datafusion-sqlparser-rs/blob/ed416548dcfe4a73a3240bbf625fb9010a4925c8/CHANGELOG.md > > - > > - > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > For additional commands, e-mail: dev-h...@datafusion.apache.org > >
Re: [VOTE] Release Apache DataFusion Ballista 44.0.0 RC1
+1 non-binding Ran `./dev/release/verify-release-candidate.sh 44.0.0 1` on M1 Mac. Ran the standalone example, `cargo run --example standalone_sql --features="ballista/standalone"` Best, Kevin Liu On Sat, Mar 1, 2025 at 7:27 AM Andy Grove wrote: > Hi, > > I would like to propose a release of Apache DataFusion Ballista version > 44.0.0. > > This release candidate is based on commit: > fc5a99c2e9fb52ae06127fa0fe2f19d6ff9406a6 [1] > The proposed release tarball and signatures are hosted at [2]. > The changelog is located at [3]. > > Please download, verify checksums and signatures, run the unit tests, and > vote > on the release. The vote will be open for at least 72 hours. > > Only votes from PMC members are binding, but all members of the community > are > encouraged to test the release and vote with "(non-binding)". > > The standard verification procedure is documented at > > https://github.com/apache/datafusion-ballista/blob/main/dev/release/README.md#verifying-release-candidates > . > > [ ] +1 Release this as Apache DataFusion Ballista 44.0.0 > [ ] +0 > [ ] -1 Do not release this as Apache DataFusion Ballista 44.0.0 because... > > My vote is +1 > > [1]: > > https://github.com/apache/datafusion-ballista/tree/fc5a99c2e9fb52ae06127fa0fe2f19d6ff9406a6 > [2]: > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-ballista-44.0.0-rc1 > [3]: > > https://github.com/apache/datafusion-ballista/blob/fc5a99c2e9fb52ae06127fa0fe2f19d6ff9406a6/CHANGELOG.md >
Re: [VOTE] Release Apache DataFusion 46.0.0 RC2
Hey folks,
I'm having issues with one particular test. It could be my local machine. I
tried restarting my laptop and running the test in isolation (`cargo test
-p datafusion --test fuzz`) but the error still persists.
```
./dev/release/verify-release-candidate.sh 46.0.0 2
failures:
fuzz_cases::sort_fuzz::test_sort_10k_mem stdout
thread 'fuzz_cases::sort_fuzz::test_sort_10k_mem' panicked at
datafusion/core/tests/fuzz_cases/sort_fuzz.rs:272:63:
called `Result::unwrap()` on an `Err` value: Execution("Failed to create
partition file at
\"/var/folders/f1/3_vzsn7x1jq9hszb3z9y6f0mgn/T/.tmpYjVAvj/.tmph79STL\":
Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
note: run with `RUST_BACKTRACE=1` environment variable to display a
backtrace
failures:
fuzz_cases::sort_fuzz::test_sort_10k_mem
test result: FAILED. 64 passed; 1 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 74.07s
```
Best,
Kevin Liu
On Tue, Mar 4, 2025 at 7:21 AM L. C. Hsieh wrote:
> +1 (binding)
>
> Verified on M4 Mac.
>
> Thanks Xudong.
>
> On Tue, Mar 4, 2025 at 7:02 AM Andrew Lamb wrote:
> >
> > +1 (binding)
> >
> > I ran the verification script on an M3 mac. Thank you so much Xudong!
> >
> > Andrew
> >
> > On Tue, Mar 4, 2025 at 9:47 AM Wang Xudong
> wrote:
> >
> > > Hi,
> > >
> > > I would like to propose a release of Apache DataFusion version 46.0.0.
> > >
> > > This release candidate is based on commit:
> > > d5ca8307940c1a6345419a2c8d91ef87704659be [1]
> > > The proposed release tarball and signatures are hosted at [2].
> > > The changelog is located at [3].
> > >
> > > Please download, verify checksums and signatures, run the unit tests,
> and
> > > vote
> > > on the release. The vote will be open for at least 72 hours.
> > >
> > > Only votes from PMC members are binding, but all members of the
> community
> > > are
> > > encouraged to test the release and vote with "(non-binding)".
> > >
> > > The standard verification procedure is documented at
> > >
> > >
> https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates
> > > .
> > >
> > > [ ] +1 Release this as Apache DataFusion 46.0.0
> > > [ ] +0
> > > [ ] -1 Do not release this as Apache DataFusion 46.0.0 because...
> > >
> > > Here is my vote:
> > >
> > > +1
> > >
> > > [1]:
> > >
> > >
> https://github.com/apache/datafusion/tree/d5ca8307940c1a6345419a2c8d91ef87704659be
> > > [2]:
> > >
> > >
> https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-46.0.0-rc2
> > > [3]:
> > >
> > >
> https://github.com/apache/datafusion/blob/d5ca8307940c1a6345419a2c8d91ef87704659be/CHANGELOG.md
> > >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org
> For additional commands, e-mail: dev-h...@datafusion.apache.org
>
>
Re: [VOTE] Release Apache DataFusion Comet 0.6.0 RC1
Ah that makes sense, thanks for following up. I see we sent the last messages within minutes of each other. Thank you for running the release! Best, Kevin Liu On Wed, Feb 19, 2025 at 2:43 PM Andy Grove wrote: > Hi Kevin, > > I think you were just unlucky with your timing and ran this after the > release candidate had been deleted from that repo (we delete the release > candidate once the release has been published). > > Thanks for trying the verification process. > > Andy. > > On Mon, Feb 17, 2025 at 12:30 PM Kevin Liu wrote: > > > Hey folks, > > > > First time running release verification for datafusion-comet. I'm > following > > the instructions here > > > > > https://github.com/apache/datafusion-comet/blob/main/dev/release/verifying-release-candidates.md > > , > > and running on the latest main branch. > > > > The script was not able to download the artifact from `dist.apache.org`. > > In > > fact, I don't see anything related to datafusion-comet under ` > > https://dist.apache.org/repos/dist/dev/datafusion/` > <https://dist.apache.org/repos/dist/dev/datafusion/> > > <https://dist.apache.org/repos/dist/dev/datafusion/> > > > > Here's the log: > > ``` > > ➜ datafusion-comet git:(main) ./dev/release/verify-release-candidate.sh > > 0.6.0 1 > > ... > > + fetch_archive apache-datafusion-comet-0.6.0 > > + local dist_name=apache-datafusion-comet-0.6.0 > > + download_rc_file apache-datafusion-comet-0.6.0.tar.gz > > + download_dist_file > > apache-datafusion-comet-0.6.0-rc1/apache-datafusion-comet-0.6.0.tar.gz > > + curl --silent --show-error --fail --location --remote-name > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-comet-0.6.0-rc1/apache-datafusion-comet-0.6.0.tar.gz > > curl: (22) The requested URL returned error: 404 > > + cleanup > > + '[' no = yes ']' > > + echo 'Failed to verify release candidate. See > > > > > /var/folders/f1/3_vzsn7x1jq9hszb3z9y6f0mgn/T/datafusion-comet-0.6.0.X.3RlR7ejtd9 > > for details.' > > Failed to verify release candidate. See > > > > > /var/folders/f1/3_vzsn7x1jq9hszb3z9y6f0mgn/T/datafusion-comet-0.6.0.X.3RlR7ejtd9 > > for details. > > ``` > > > > Please let me know if I'm missing something. > > > > Best, > > Kevin Liu > > > > On Fri, Feb 14, 2025 at 12:21 PM L. C. Hsieh wrote: > > > > > +1 (binding) > > > > > > Verified on M4 Mac. > > > > > > Thanks Andy. > > > > > > On Fri, Feb 14, 2025 at 11:27 AM Andy Grove > > wrote: > > > > > > > > Hi, > > > > > > > > I would like to propose a release of Apache DataFusion Comet version > > > 0.6.0. > > > > > > > > This release candidate is based on commit: > > > > b735f2debb983e96d8f4a8a948ef4d61f80f6bcb [1] > > > > The proposed release tarball and signatures are hosted at [2]. > > > > Pre-built jar files are available in a Maven staging repository [3]. > > > > The changelog is located at [4]. > > > > > > > > Please download, verify checksums and signatures, run the unit tests, > > and > > > > vote > > > > on the release. The vote will be open for at least 72 hours. > > > > > > > > Only votes from PMC members are binding, but all members of the > > community > > > > are > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > The standard verification procedure is documented at > > > > > > > > > > https://github.com/apache/datafusion-comet/blob/main/dev/release/verifying-release-candidates.md > > > > > > > > [ ] +1 Release this as Apache DataFusion Comet 0.6.0 > > > > [ ] +0 > > > > [ ] -1 Do not release this as Apache DataFusion Comet 0.6.0 > because... > > > > > > > > Here is my vote: > > > > > > > > +1 > > > > > > > > [1]: > > > > > > > > > > https://github.com/apache/datafusion-comet/tree/b735f2debb983e96d8f4a8a948ef4d61f80f6bcb > > > > [2]: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-comet-0.6.0-rc1 > > > > [3]: > > > > https://repository.apache.org/#nexus-search;quick~org.apache.datafusion > > > > [4]: > > > > > > > > > > https://github.com/apache/datafusion-comet/blob/b735f2debb983e96d8f4a8a948ef4d61f80f6bcb/CHANGELOG.md > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > > > For additional commands, e-mail: dev-h...@datafusion.apache.org > > > > > > > > >
Re: [RESULT] [VOTE] Release DataFusion Python Bindings 45.2.0 RC2
I see datafusion 45.2.0 on pypi now, https://pypi.org/project/datafusion/ Thanks for working on the release everyone! :) Best, Kevin Liu On Sun, Feb 23, 2025 at 5:11 AM Andrew Lamb wrote: > With 5 +1 votes (3 binding) the release is approved. > > The release is available here: > > > https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-45.2.0 > > I have also uploaded the release to crates.io > https://crates.io/crates/datafusion-python/45.2.0 > > Andrew > > > On Wed, Feb 19, 2025 at 12:38 PM Andy Grove wrote: > > > +1 (binding) > > > > Verified on Ubuntu 22.04.5 > > > > Thanks, Tim! > > > > I did see a bunch of warnings, btw: > > > > warning: use of deprecated associated function > > `pyo3::types::PyModule::new_bound`: renamed to `PyModule::new` > >--> src/lib.rs:111:27 > > | > > 111 | let store = PyModule::new_bound(py, "object_store")?; > > > > On Wed, Feb 19, 2025 at 5:26 AM Tim Saucer wrote: > > > > > Thank you, Kevin! > > > > > > On Wed, Feb 19, 2025 at 12:59 AM Kevin Liu > > wrote: > > > > > > > Hey folks, > > > > > > > > Following up on the conversation about pyarrow, v19.0.1 has just been > > > > released on pypi, https://pypi.org/project/pyarrow/. > > > > I reran the test suite using `pyarrow==19.0.1` and verified that all > > > tests > > > > passed. > > > > > > > > Best, > > > > Kevin Liu > > > > > > > > On Tue, Feb 18, 2025 at 3:04 AM Andrew Lamb > > > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > I ran the release verification script and it worked well for me on > a > > M3 > > > > mac > > > > > > > > > > Thanks Tim and everyone else who is verifying these releases (Kevin > > it > > > is > > > > > great to see you here, and L.C. your voting participation is > > amazing!) > > > > > > > > > > Andrew > > > > > > > > > > On Mon, Feb 17, 2025 at 8:01 PM Kevin Liu > > > wrote: > > > > > > > > > > > +1 (non-binding) Thanks for running this release! > > > > > > > > > > > > I was able to run the verification script successfully on M1 Mac. > > > > > > ``` > > > > > > ./dev/release/verify-release-candidate.sh 45.2.0 2 > > > > > > ``` > > > > > > > > > > > > I also verified by running pytest directly, based on > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-a-release > > > > > > , > > > > > > after downgrading to `pyarrow==18.0.0` > > > > > > ``` > > > > > > git fetch --tags > > > > > > git checkout 45.2.0-rc2 > > > > > > git submodule update --init --recursive > > > > > > python3 -m venv .venv > > > > > > source .venv/bin/activate > > > > > > pip install --extra-index-url https://test.pypi.org/simple/ > > > > > > datafusion==45.2.0 > > > > > > pip install pytest > > > > > > pip install pyarrow==18.0.0 > > > > > > pytest --import-mode=importlib python/tests > > > > > > ``` > > > > > > > > > > > > For context, here's the error I encountered using > `pyarrow=19.0.0`, > > > > > > > > https://gist.github.com/kevinjqliu/6d111bf7bd74c165c34e4366fe53a6c6. > > > > > > Looks like we don't pin a specific pyarrow version, so when the > new > > > > > version > > > > > > (`19.0.1`) is released next week, the issue should be > automatically > > > > > > resolved. > > > > > > > > > > > > Best, > > > > > > Kevin Liu > > > > > > > > > > > > On Mon, Feb 17, 2025 at 3:32 PM L. C. Hsieh > > > wrote: > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > Verified on M4 Mac. > > > > > > > > > > > > > > Thanks Tim. > > > > > >
Re: Blog Post for 45 Release
Linking back to the devlist https://datafusion.apache.org/blog/2025/02/20/datafusion-45.0.0/ Thanks again for the great blog! Best, Kevin Liu On Sun, Feb 23, 2025 at 5:16 AM Andrew Lamb wrote: > Hello DataFusion community. > > Bruce has created a blog post for our most recent release[1]. > > It looks quite good in my opinion and I plan to merge / publish it > tomorrow. Please leave any comments you may have or let us know if you > would like more time to review > > Andrew > > [1]: https://github.com/apache/datafusion-site/pull/57 >
Re: [VOTE] Release DataFusion Python Bindings 47.0.0 RC1
+1 (non-binding) Verified both using the Verifying a Release instruction on GitHub <https://github.com/apache/datafusion-python/blob/main/dev/release/README.md#verifying-a-release> and the verification script (`./dev/release/verify-release-candidate.sh 47.0.0 1`) Thanks for working on the release! Best, Kevin Liu On Mon, May 19, 2025 at 10:41 AM Andrew Lamb wrote: > +1 (binding) > > Verified on an M3 Mac > > Thank you Tim > > > > On Mon, May 19, 2025 at 1:02 PM L. C. Hsieh wrote: > > > +1 (binding) > > > > Verified on M4 Mac. > > > > Thanks Tim. > > > > > > On Mon, May 19, 2025 at 9:54 AM Tim Saucer wrote: > > > > > > Hi, > > > > > > I would like to propose a release of the Apache DataFusion Python > > Bindings, > > > version 47.0.0. > > > > > > This release candidate is based on commit: > > > ebd62d052cd54fccbcf73fedab8ed042752c1dd6 [1] > > > The proposed release tarball and signatures are hosted at [2]. > > > The changelog is located at [3]. > > > The Python wheels are located at [4]. > > > > > > Please download, verify checksums and signatures, run the unit tests, > and > > > vote > > > on the release. The vote will be open for at least 72 hours. > > > > > > Only votes from PMC members are binding, but all members of the > community > > > are > > > encouraged to test the release and vote with "(non-binding)". > > > > > > The standard verification procedure is documented at > > > https://github.com/apache/d > > > > > > atafusion-python/blob/main/dev/release/README.md#verifying-release-candidates. > > > > > > [ ] +1 Release this as Apache DataFusion Python 47.0.0 > > > [ ] +0 > > > [ ] -1 Do not release this as Apache DataFusion Python 47.0.0 > because... > > > > > > Here is my vote: > > > > > > +1 (non-binding) > > > > > > [1]: > > > > > > https://github.com/apache/datafusion-python/tree/ebd62d052cd54fccbcf73fedab8ed042752c1dd6 > > > [2]: > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-47.0.0-rc1 > > > [3]: > > > > > > https://github.com/apache/datafusion-python/blob/ebd62d052cd54fccbcf73fedab8ed042752c1dd6/CHANGELOG.md > > > [4]: https://test.pypi.org/project/datafusion/47.0.0/ > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > > For additional commands, e-mail: dev-h...@datafusion.apache.org > > > > >
Re: [ANNOUNCE] New DataFusion PMC member: Xudong Wang (xudong963)
Congratulations!! 🥳 On Sat, May 24, 2025 at 10:13 AM Xuanwo wrote: > Congrats, xudong! > > On Sat, May 24, 2025, at 18:48, Andrew Lamb wrote: > > The Project Management Committee (PMC) for Apache DataFusion has invited > > $Xudong to become a PMC member and we are pleased to announce > > that he has accepted. > > > > Congratulations and welcome! > > -- > Xuanwo > > https://xuanwo.io/ > > - > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > For additional commands, e-mail: dev-h...@datafusion.apache.org > >
Re: [ANNOUNCE] New DataFusion committer: Adrian Garcia Badaracco
Congratulations! On Thu, May 22, 2025 at 4:17 AM Tim Saucer wrote: > Congratulations Adrian! > > > On Wed, May 21, 2025 at 9:17 PM Phillip LeBlanc > wrote: > > > Congrats Adrian! > > > > From: Andrew Lamb > > Date: Thursday, May 22, 2025 at 6:34 AM > > To: dev@datafusion.apache.org > > Subject: [ANNOUNCE] New DataFusion committer: Adrian Garcia Badaracco > > > > On behalf of the DataFusion PMC, I'm happy to announce that Adrian > Garcia > > Badaracco has accepted an invitation to become a committer on Apache > > DataFusion. Welcome, and thank you for your contributions! > > > > Andrew > > >
Re: [VOTE] Release Apache DataFusion 48.0.0 RC3
+1 (non-binding) Ran the RC verification script and a few local tests. Best, Kevin Liu On Sun, Jun 8, 2025 at 12:45 PM Dewey Dunnington wrote: > Thank you all for considering a final addition! My vote is +1 > (non-binding). > > I ran dev/release/verify-release-candidate.sh 48.0.0 3 on MacOS 15.5 (M4). > > Cheers, > > -dewey > > > > On Sat, Jun 7, 2025 at 11:56 AM Andy Grove wrote: > > > +1 (binding) > > > > Verified by upgrading Comet in > > https://github.com/apache/datafusion-comet/pull/1863 > > > > Thanks Xudong! > > > > On Sat, Jun 7, 2025 at 9:35 AM L. C. Hsieh wrote: > > > > > +1 (binding) > > > > > > Verified on M4 Mac. > > > > > > Thanks Xudong. > > > > > > On Fri, Jun 6, 2025 at 11:39 PM Wang Xudong > > wrote: > > > > > > > > Hi, > > > > > > > > I would like to propose a release of Apache DataFusion version > 48.0.0. > > > > > > > > This release candidate is based on commit: > > > > 33a32d4382bee7e3c705d0f55d05c24a115a2f98 [1] > > > > The proposed release tarball and signatures are hosted at [2]. > > > > The changelog is located at [3]. > > > > > > > > Please download, verify checksums and signatures, run the unit tests, > > and > > > > vote > > > > on the release. The vote will be open for at least 72 hours. > > > > > > > > Only votes from PMC members are binding, but all members of the > > community > > > > are > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > The standard verification procedure is documented at > > > > > > > > > > https://github.com/apache/datafusion/blob/main/dev/release/README.md#verifying-release-candidates > > > > . > > > > > > > > [ ] +1 Release this as Apache DataFusion 48.0.0 > > > > [ ] +0 > > > > [ ] -1 Do not release this as Apache DataFusion 48.0.0 because... > > > > > > > > Here is my vote: > > > > > > > > +1 binding > > > > > > > > [1]: > > > > > > > > > > https://github.com/apache/datafusion/tree/33a32d4382bee7e3c705d0f55d05c24a115a2f98 > > > > [2]: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-48.0.0-rc3 > > > > [3]: > > > > > > > > > > https://github.com/apache/datafusion/blob/33a32d4382bee7e3c705d0f55d05c24a115a2f98/CHANGELOG.md > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > > > For additional commands, e-mail: dev-h...@datafusion.apache.org > > > > > > > > >
Re: [ANNOUNCE] New DataFusion committer: Siew Kam Onn (kosiew)
Congratulations! On Mon, Jun 16, 2025 at 4:31 AM Tim Saucer wrote: > Congratulations! Very well deserved! > > On Mon, Jun 16, 2025 at 7:19 AM Andrew Lamb > wrote: > > > On behalf of the DataFusion PMC, I'm happy to announce that Siew Kam > > Onn (kosiew) > > has accepted an invitation to become a committer on Apache > > DataFusion. Welcome, and thank you for your contributions! > > >
Re: [ANNOUNCE] New DataFusion PMC member: Tim Saucer (timsaucer)
Well deserved, congratulations!! Best, Kevin Liu On Mon, Jun 16, 2025 at 8:43 AM Trevor Barnes wrote: > Congrats, Tim! Thanks for all of your help and support > > Trevor Barnes > trevor.barne...@gmail.com > 502-379-5130 > > > On Jun 16, 2025, at 8:02 AM, Dewey Dunnington < > dewey.dunning...@gmail.com> wrote: > > > > Congratulations! > > > >> On Mon, Jun 16, 2025 at 7:08 AM Marko Milenković < > milenkovi...@gmail.com> > >> wrote: > >> > >> Congratulations Tim! > >> -- > >> Marko Milenković > >> > >> > >> > >> > >>>> On 16 Jun 2025, at 12:18, Andrew Lamb wrote: > >>> > >>> The Project Management Committee (PMC) for Apache DataFusion has > invited > >>> Tim Saucer (timsaucer) to become a PMC member and we are pleased to > >> announce > >>> that Tim has accepted. > >>> > >>> Congratulations and welcome! > >> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > >> For additional commands, e-mail: dev-h...@datafusion.apache.org > >> > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@datafusion.apache.org > For additional commands, e-mail: dev-h...@datafusion.apache.org > >
