Hi all, Over the past year, there's been a lot of discussion around the challenges we face as a project in doing releases. Because they are costly to do, we don't do them often; because we don't do them often, they become even costlier.
There are only a small number of people (PMC members with GPG keys registered with ASF) who could possibly be release manager, and because of the amount of time required (I saw Krisztián say on the 3.0 release thread something like "I'll start a new rc, it'll be done in 12 hours), even fewer people could be expected to take on the burden. Indeed, this is Krisztián's 10th release in a row as release manager, and over the course of the project, 2/3 of all release candidates have been made by just 2 people. I'd like to propose a change to our release procedure: instead of having the release candidate vote include Python wheels, Linux system packages, or any other binary packages, we should only vote on the source release. Binary artifacts would be produced as post-release tasks, using the official source release. This would greatly reduce the time and effort it takes to produce a release candidate--tar, sign, and upload, that's it--and it would remove a bunch of points of failure from the release-candidate making process (timeouts, CI flakiness, etc.). It would also mean fewer release-blocking issues--we still have to fix the packaging builds, but doing so can happen in parallel with the verification process. If we found problems in the packaging scripts, fixes could either be applied as patch steps to the binary artifact build scripts, or if fixes can be produced quickly, we collect them and cut another (cheap) release candidate. Right now, our only option is the latter, which makes for a slow, stressful release process where there are so many places where a simple issue can block the whole release or set us back an additional week (a full day to produce a release candidate plus another three to vote). If we went this direction, we could still choose to vote separately on binary packages like wheels, though I'm not sure that's worth the effort. Many of the packages that people use (conda, homebrew, CRAN, etc.) are already "unofficial" releases because they're packaged by someone else, and I don't think the distinction is meaningful to our users. To be clear, this doesn't reduce the general maintenance burden of the project. We still have to monitor nightly builds, fix packaging scripts that break, and deal with CI service interruptions. This change would just reduce the burden on the release manager and allow us to spread more broadly the costs of packaging and releasing. It also solves questions such as "Why should the Rust release be blocked just because we're having a problem building Python wheels on macOS?" There are also other things we could do that would, on a technical level, improve our ability to make releases more efficiently. Andy Grove's change in the use of maven in the release process will help, as would a number of CI/CD improvements. I view these as complementary to this proposal, which is a governance question with technical/logistical implications. Thoughts? Neal
