Quoting Stefan Hajnoczi (2021-10-04 04:01:22) > On Fri, Oct 01, 2021 at 09:39:13AM +0200, Philippe Mathieu-Daudé wrote: > > On 9/30/21 15:40, Stefan Hajnoczi wrote: > > > Hi Mike, > > > QEMU downloads are currently hosted on qemu.org's Apache web server. > > > Paolo and I were discussing ways to reduce qemu.org network traffic to > > > save money and eventually turn off the qemu.org server since there is no > > > full-time sysadmin for it. I'd like to discuss moving QEMU downloads to > > > GitLab Releases. > > > > > > Since you create and sign QEMU releases I wanted to see what you think > > > about the idea. GitLab Releases has two ways of creating release assets: > > > archiving a git tree and attaching arbitrary binaries. The > > > scripts/make-release script fetches submodules and generates version > > > files, so it may be necessary to treat QEMU tarballs as arbitrary > > > binaries instead of simply letting GitLab create git tree archives: > > > https://docs.gitlab.com/ee/user/project/releases/#use-a-generic-package-for-attaching-binaries > > > > > > Releases can be uploaded via the GitLab API from your local machine or > > > deployed as a GitLab CI job. Uploading from your local machine would be > > > the closest to the current workflow. > > > > > > In the long term we could have a CI job that automatically publishes > > > QEMU releases when a new qemu.git tag is pushed. The release process > > > could be fully automated so that manual steps are no longer necessary, > > > although we'd have to trust GitLab with QEMU GPG signing keys. > > > > Before having to trust a SaaS for GPG signing, could this work? > > > > - make-release script should produce a reproducible tarball in a > > gitlab job, along with a file containing the tarball hash. > > > > - Mike (or whoever is responsible of releases) keeps doing a local > > manual build > > > > - Mike checks the local hash matches the Gitlab artifact hash > > > > - Mike signs its local build/hash and uses the GitLab API to upload > > that .sig to job artifacts > > > > - we can have an extra manual job that checks the tarball.sig > > (hash and pubkey) and on success deploys updating the download > > page, adding the new release > > I wonder what Mike sees as the way forward.
Hi Stefan, Philippe, In general I like the idea, since we could also have the CI do the full gamut of testing against the binaries built from said tarball, so the Release Person can just regenerate the tarball and provide a sig to attest that it came from the proper sources. Currently I do make check and make check-acceptance and a few other sanity checks, which I guess would no longer be needed then. But I think the more immediate issue is where/how to host those tarballs. Even moving all the ROMs/capstone out of the source tree still results in an xz-compressed tarball size ~25MB, which is well above the 10MB limit mentioned earlier. We could break it out into target-specific tarballs, maybe further into softmmu/user variants, but that sounds painful for both users and maintainers who need to deal with the resulting source tree differences. What I'm wondering is whether we could just use the archive files generated by gitlab when we tag our releases? E.g.: https://gitlab.com/qemu-project/qemu/-/archive/v6.1.0/qemu-v6.1.0.tar.bz2 If we paired that with an in-tree script similar to make-release for users to download individual ROM sources/subprojects used for a particular tagged release, would that be sufficient for GPL compliance and verifying what sources the binaries were built from? Are there any other considerations WRT ROMs/etc.? With something like that in place, Release Person could just do a git checkout, verify the Maintainer's sig/tag (in case we don't necessarily trust the git host), generate the tarball, verify the hash matches what gitlab published (or verify/diff individual files if the bz/gz hashes require a specific environment), then sign the gitlab tarball and add the sig to qemu.org download page along with a link the gitlab-generated tarball. We could also publish the Maintainer and Release Person public keys on qemu.org download page so users can verify this as well using the same process. Users that want the additional sources can then do it locally via above-mentioned script, which would be part of the now-signed tarball and so could be 'trusted' assuming the individual project hosts weren't compromised (which is still an assumption that's needed with the current process anyway). I guess the main question is who is using the ROM/BIOS sources in the tarballs, and would this 2-step process work for those users? If there are distros relying on them then maybe there are some more logistics to consider since the make-release downloads are both time-consuming and prone to network errors/stalls since it relies on the availability of a good number of different hosts. -Mike > > Stefan
