Daniel and I had a chat about improving our release process today and I'd like to make some proposals.
1) Decrease the amount of testing done by the every commit buildbots on the branches. This bots should be a quick sanity check, does it build and does it pass basic tests. 2) Create a buildbot setup that generates a tarball on each branch nightly (if there were any changes). It would do this in the same process that a RM would use (and eventually would become the process for producing the tarball). 3) This tarball would then have various build bots run tests off it. These tests would be much more extensive. In particular it would run cross version (SERVER_MINOR_VERSION), packing, sharding and other variations of our test suite that frankly probably don't get as much attention now as they should. 4) Eventually make it so that the RM asks the buildbot to produce the release tarball, including running the nightly buildbots against it. RM then downloads the tarball, compares it to the branch@revision for differences looking for any unexpected diffs, and then signs it. It gets put up for everyone to approve. 5) Humans would no longer do much testing. We'd have buildbots for the major platforms, which would have been included by our manual testing from the past. The build bots would be running far more extensive tests than we ever would be likely to run on our own. So the only testing we'd need to do is verifying that the builds match the source we expect and whatever sanity testing we felt like we wanted to do. This should lower the bar dramatically for getting tests/signatures out. Humans don't need to do this boring and repetitive work. It should let everyone spend more time on more important issues than manual testing. Infra had offered us some more buildbot resources here recently. This seems like the logical way to use them.
