Hey Kevin, First of all, thanks for working on the releases, that's always much appreciated.
Regarding the changes to the release process, I'm all for automating as much as possible, but I have some concerns. I also think it is important to split out nightly builds, and the release process in general. Releases Concerning the releases, I think the official ASF release policy <https://www.apache.org/legal/release-policy.html#artifacts> is a very good read. While reading up on this topic, I noticed that it is allowed to have automated signing of the release <https://infra.apache.org/release-signing.html#automated-release-signing>, but this comes with some prerequisites, such as having reproducible builds. This is not the case for PyIceberg today, ie. if you build wheels twice, they have different checksums. Also, a manual validation step is still part of the process, where all artifacts are produced on trusted hardware <https://www.apache.org/legal/release-policy.html#owned-controlled-hardware> before publication. I would lean much more towards the way Iceberg-Go has solved this <https://github.com/apache/iceberg-go/blob/main/dev/release/release_rc.sh>. It creates a tag locally and pushes it to the repository, the tag triggers a Github Action workflow, generating the required artifacts and the convenience artifacts. Those are downloaded to the local machine, signature, and checksum are added, and pushed back to GitHub Actions and SVN. Nightly Builds As also stated in the release policy. We can provide nightly builds for the development community, but as a project, we should direct outsiders toward the official releases. Since a nightly build is not an official release <https://repository.apache.org/content/groups/snapshots/org/apache/iceberg/>, it doesn't have to go through the whole process including signatures and checksums, as long it is sufficiently hidden from the end-users and intended for developers only. We could, for example, push a nightly build to test.pypi.org <https://test.pypi.org/project/pyiceberg/>. Thanks again for working on this. Hope this helps, and let me know what you think! Kind regards, Fokko Op ma 2 dec 2024 om 20:38 schreef Kevin Liu <kevinjq...@apache.org>: > Hi everyone, > > As the release manager for PyIceberg 0.8.0 and the upcoming 0.8.1 release, > I’ve taken some time to reflect on ways we could improve the release > process. I drew inspiration from the iceberg-go release process and > documented my notes here > <https://github.com/apache/iceberg-python/issues/1306>. I’ve also updated > the release instructions here > <https://py.iceberg.apache.org/how-to-release/>. > > Currently, the release process is manual and prone to errors. My goal is > to automate it as much as possible, ideally transforming it into a > single-click process. > > I’d like to gather your thoughts on two key ideas: > > 1. Automating the release process to reduce manual steps and errors. > 2. Introducing nightly builds to PyPI once automation is in place (issue > #872 <https://github.com/apache/iceberg-python/issues/872>). > > The PyIceberg release process can be summarized in these steps: > > - Create a Release Candidate (RC) > - Vote on the devlist > - Promote the RC to a Final Release > > I believe the *"*Create a Release Candidate*"* step can benefit the most > from automation. Here’s a breakdown of the current steps: > > - Create a tag for the Release Candidate (e.g., `0.8.1rc1`). > - Generate artifacts (currently done using GitHub Actions). > - Generate SHA-512 checksums and GPG signatures, then upload the > artifacts to SVN. > - Upload the artifacts to PyPI. > > To automate these steps via GitHub Actions, we’d need to address the > following: > > - *GPG Signing*: GitHub Actions require a `GPG_PRIVATE_KEY` secret. > I’ve tested this with my own key, but it would be better to create a new > key (possibly owned by ASF) for signing files. > - *SVN Uploads*: Uploading artifacts to SVN requires credentials. I > haven’t tested this step yet, but we should aim to use credentials provided > by ASF Infra instead of personal ones. > - *PyPI Uploads*: Similarly, uploading to PyPI requires an API token, > which should ideally be provided by ASF Infra. > > I’ve begun automating the artifact generation process (PR #1391 > <https://github.com/apache/iceberg-python/pull/1391>). However, the > release manager currently still needs to manually download and upload > artifacts to both SVN and PyPI. > > Once the "Create a Release Candidate" step is automated, we can create a > GitHub Action to manually build and upload a nightly version to PyPi. > > > *Is this the direction we want to take for the release process? If so, > what’s the best way to coordinate with ASF Infra to create the necessary > credentials?* > > I’d love to hear your thoughts and any additional suggestions. > Best, > Kevin Liu > >
