I'm OK with moving to source only releases, but we need to take a step back and consider how our CI/CD is failing to notify us in a suitably timely and automated way about the packages being broken. For example, the fact that we had 2 failed RCs as the result of packaging issues points to a broken process.
So there are a couple issues at play: * The act of _producing_ the package artifacts should not stop a release vote from proceeding like it does now (the "12 hours" you refer to that's caused by slow iteration time with Crossbow — this is also a problem, can we not fix this?) * We need a better feedback loop to determine whether master is in a releasable state, including all relevant packages If we commit ourselves to solving one problem but not both, I fear that we will find ourselves suffering from other kinds of problems in future release cycles On Tue, Jan 19, 2021 at 5:16 PM Neal Richardson <neal.p.richard...@gmail.com> wrote: > > Hi all, > Over the past year, there's been a lot of discussion around the challenges > we face as a project in doing releases. Because they are costly to do, we > don't do them often; because we don't do them often, they become even > costlier. > > There are only a small number of people (PMC members with GPG keys > registered with ASF) who could possibly be release manager, and because of > the amount of time required (I saw Krisztián say on the 3.0 release thread > something like "I'll start a new rc, it'll be done in 12 hours), even fewer > people could be expected to take on the burden. Indeed, this is Krisztián's > 10th release in a row as release manager, and over the course of the > project, 2/3 of all release candidates have been made by just 2 people. > > I'd like to propose a change to our release procedure: instead of having > the release candidate vote include Python wheels, Linux system packages, or > any other binary packages, we should only vote on the source release. > Binary artifacts would be produced as post-release tasks, using the > official source release. > > This would greatly reduce the time and effort it takes to produce a release > candidate--tar, sign, and upload, that's it--and it would remove a bunch of > points of failure from the release-candidate making process (timeouts, CI > flakiness, etc.). It would also mean fewer release-blocking issues--we > still have to fix the packaging builds, but doing so can happen in parallel > with the verification process. If we found problems in the packaging > scripts, fixes could either be applied as patch steps to the binary > artifact build scripts, or if fixes can be produced quickly, we collect > them and cut another (cheap) release candidate. Right now, our only option > is the latter, which makes for a slow, stressful release process where > there are so many places where a simple issue can block the whole release > or set us back an additional week (a full day to produce a release > candidate plus another three to vote). > > If we went this direction, we could still choose to vote separately on > binary packages like wheels, though I'm not sure that's worth the effort. > Many of the packages that people use (conda, homebrew, CRAN, etc.) are > already "unofficial" releases because they're packaged by someone else, and > I don't think the distinction is meaningful to our users. > > To be clear, this doesn't reduce the general maintenance burden of the > project. We still have to monitor nightly builds, fix packaging scripts > that break, and deal with CI service interruptions. This change would just > reduce the burden on the release manager and allow us to spread more > broadly the costs of packaging and releasing. It also solves questions such > as "Why should the Rust release be blocked just because we're having a > problem building Python wheels on macOS?" > > There are also other things we could do that would, on a technical level, > improve our ability to make releases more efficiently. Andy Grove's change > in the use of maven in the release process will help, as would a number of > CI/CD improvements. I view these as complementary to this proposal, which > is a governance question with technical/logistical implications. > > Thoughts? > > Neal
