On 03/08/19 14:42, Daniel P. Berrangé wrote: > On Fri, Mar 08, 2019 at 02:09:53PM +0100, Laszlo Ersek wrote: >> Hi, >> >> I have a mostly-ready-for-posting patch set for $SUBJECT. My question is >> what QEMU release I should be targeting with it. >> >> The Soft Feature Freeze for 4.0 is on 2019-03-12. Here's why that's a >> bit inconvenient for me. >> >> The upcoming EDK2 stable release is edk2-stable201903, and it is planned >> for... today. >> >> >> https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Release-Planning#edk2-stable201903-tag-planning >> >> But, it's being blocked (at least one CVE fix still needs merging, but >> there could be something else too). I don't know what that will mean for >> the actual tag date. Maybe next Monday (the 11th)? >> >> In my series, I'd like to advance QEMU's roms/edk2 submodule to this new >> release. But that might leave us with 1 day before the QEMU 4.0 Soft >> Feature Freeze (see above), i.e., for me to post the series, and for a >> submaintainer to send a pullreq with it. That's a bit too tight. > > IMHO if you advanced the submodule hash to a nearly-released version > before freeze, it would be fine to then advance it again to the actually > released commit hash during QEMU freeze, because presumably the EDK > changes are similarly bugfix only at this point in its release process.
Great idea! Thanks! And, indeed, I had shamelessly stolen the SFF and HFF concepts from QEMU, when I proposed them for edk2. Edk2 doesn't have maintainer trees and pull requests, so I tried to "adopt" the SFF and HFF definitions. We should likely improve those over time, as the edk2 workflow too (hopefully!) matures. https://github.com/tianocore/tianocore.github.io/wiki/SoftFeatureFreeze https://github.com/tianocore/tianocore.github.io/wiki/HardFeatureFreeze >> I'm not in a mortal rush to get this into 4.0, but the next release >> cycles (in three months, approximately?) might align similarly, between >> edk2 and QEMU. It would be best to avoid QEMU carrying edk2 platform >> firmware that is at all times at least three months old. The main reason >> is that CVEs tend to exist, for both edk2 proper, and for the specific >> OpenSSL release that is bundled with the given edk2 stable tag. And edk2 >> doesn't yet have stable *branches*. >> >> Should we try to squeeze my set into 4.0 (possibly moving the Soft >> Feature Freeze), or just aim for 4.1? >> >> Also, who'd be the maintainer to queue my set? I mostly thought of Gerd, >> due to his work on iPXE and SeaBIOS. Here's the current diffstat: >> >> Makefile | 17 +- >> pc-bios/README | 11 + >> pc-bios/descriptors/50-edk2-i386-secure.json | 34 +++ >> pc-bios/descriptors/50-edk2-x86_64-secure.json | 35 +++ >> pc-bios/descriptors/60-edk2-aarch64.json | 31 +++ >> pc-bios/descriptors/60-edk2-arm.json | 31 +++ >> pc-bios/descriptors/60-edk2-i386.json | 33 +++ >> pc-bios/descriptors/60-edk2-x86_64.json | 34 +++ >> pc-bios/edk2-aarch64-code.fd | Bin 0 -> 67108864 bytes >> pc-bios/edk2-arm-code.fd | Bin 0 -> 67108864 bytes >> pc-bios/edk2-arm-vars.fd | Bin 0 -> 67108864 bytes >> pc-bios/edk2-i386-code.fd | Bin 0 -> 3653632 bytes >> pc-bios/edk2-i386-secure-code.fd | Bin 0 -> 3653632 bytes >> pc-bios/edk2-i386-vars.fd | Bin 0 -> 540672 bytes >> pc-bios/edk2-licenses.txt | 209 +++++++++++++++ >> pc-bios/edk2-x86_64-code.fd | Bin 0 -> 3653632 bytes >> pc-bios/edk2-x86_64-secure-code.fd | Bin 0 -> 3653632 bytes > > > Yikes, am I really reading those sizes right ? The biggest ROMs there > are 64 MB, so this is proposing to add ~206 MB of binaries to the > pc-bios directory ? Sort of. This is because of how the "virt" machine type's pflash chips are sized. The edk2 build produces a 2MB firmware executable and a 768KB varstore template for each of aarch64 and arm, but (a) QEMU doesn't auto-pad either [*], and (b) if we used qcow2 with compression, then libvirt couldn't deal with the images [**]. Hence we have to pad the files manually, after the edk2 build completes. [*] We've just been discussing auto-padding in a parallel patch from Alex and Markus (the latest version is "[PATCH v7] pflash: Require backend size to match device, improve errors", in which the padding has already been dropped). But such padding would only work for read-only pflash mappings anyway, not for variable store files. [**] The root cause for not using qcow2 with pflash images is that qcow2 would make them candidates for "savevm" to dump live memory contents into them, which is Very Bad (TM). > I think this is a very undesirable thing to do. Yup, not ideal. > Consider that we'll need to refresh those ROMs multiple times a year to > track updates or security fixes. It will result in our .git repo size > growing massively over time. Actually, it's not *that* bad. Earlier I investigated how the formatted variant of the patch (adding these blobs) would look like. First, I compressed all the "edk2-*fd" files (individually) with "gzip --best", just to get a byte count. That yielded almost precisely 9MB, in total. Second, the formatted patch, adding these blobs, is 12.6MB. Taking the 4/3 blowup factor from base64 encoding, the "unexplained overhead" is not that huge (0.6 MB). Obviously such a patch is still impossible to post to the list, hence I'd format the series with "--no-binary" (which produces the view that you get in "gitk" too). My point is that the initial internal storage requirement, especially after git-gc / packing, would not exceed 9MB by much. There are two other mitigating factors: - "git checkout" is smart enough to punch holes into the files in the working directory. $ du -csh edk2-{arm,aarch64}-code.fd edk2-arm-vars.fd 2.1M edk2-arm-code.fd 2.1M edk2-aarch64-code.fd 772K edk2-arm-vars.fd 4.8M total This happens despite the fact that the Makefile performs the padding with "cat /dev/zero" and "head". - whenever we update binary files and format those changes to patch emails, git only formats the actual changes. Given that the padding (which covers most of these files) is virtually never expected to change, the incremental updates (which are compressed as well) shouldn't be huge. > I don't think people will like cloning > multi-GB sized repos after a few years of ROM updates. It shouldn't be more than a few MB with every update, in effect. > As I've mentioned before, I think QEMU should get out of the business > of distributing ROMs in its primary released qemu-x.x.x.tar.gz archives, > and provide them as a separate tar.gz bundle. Even better if we can > move the existing ROMS out of git too, though we have to consider how > developers biulding from git would access the ROMs & know when they > need to acquire new copies. I'm 100% fine with not bundling any firmware binaries for end-users; however I've been doing this work because Igor needs full platform firmware binaries for ACPI unit tests. In particular, in aarch64 guests, there is no ACPI without UEFI, and in UEFI guests, finding the ACPI entry point (RSD PTR) is convoluted enough that it needs dedicated guest support. The UEFI test helper application for that is already upstream (and it is really small, see 4d3f50eb489e..503bb0b975ab). This application "preps" the guest RAM so that the qtest program can find the ACPI entry point via raw memory reads. But, the UEFI helper app (which is provided as a UEFI-bootable ISO image) must be launched against a full platform firmware, from the qtest case. Originally we thought the qtest case could use firmware images right from the build host, but locating those on any particular host proved a mess, so Igor asked me to add these images. > The main important things to version control are the build config and > the git submodule version information. Gerd has been offering an external firmware repo at <https://www.kraxel.org/repos/> for years now, and the build scripts (in the RPM spec file) are tracked in a publicly cloneable git repo. I'm entirely okay with your proposal, but personally I wouldn't like to invest time into (further) developing such a service. (I've done the current work because it looked necessary after quite a bit of navel gazing for alternative approaches -- nobody else appeared enthusiastic about implementing the one approach that seemed workable.) Thanks, Laszlo
