Hi! On Fri, 2023-09-01 at 08:43:55 +1200, Michael Hudson-Doyle wrote: > Recently the topic of exploiting newer instructions without dropping > support for older machines has come up several times inside Ubuntu > engineering. I understand this topic has come up several times in the past > for Debian as well, but nothing has really come of it to date.
I also had a chat about this with Matthias Klose (CCed) around 2022-05. > I've spent a while thinking through the options and coming up with a design > and wrote some notes into a wiki page: > https://wiki.debian.org/ArchitectureVariants I think we are already doing 1, 2 and 3. I agree 4 is just wrong. And something like 5 is what I suggested to Matthias for Ubuntu when we last discussed it as the best way to go about this. I'm not sure I entirely agree with the requirements you set forth though: - I think such optimized builds might need to be done with "special toolchains" (these could simply be wrappers over the host compiler passing the appropriate flags via command-line or via specs or similar, not necessarily full blown toolchains), passing these via something like dpkg-buildflags seems currently unreliable, as I don't think we have full coverage in packages (neither for all compilers available)? Although it would be better as it would centralize the management. (For reference this is in part how rpm handles this: https://github.com/rpm-software-management/rpm/blob/master/rpmrc.in) - Perhaps that's a limitation from the archive software side, but requiring to place the binary packages in the same pool seems rather restrictive (it forces different filenames for example). - I guess it might be nice for the ISA to be passed down to the dpkg tools, but I don't think this is strictly necessary? A frontend like apt could also decide based on metadata in say the Release file, although not having the actual installed package metadata on whether it was a different ISA build or not would make its job more inconvenient. In any case I don't have a big issue with recording this via dpkg-gencontrol or similar if necessary. On the specific implementation details: - Changing the Architecture format (as in adding colons there) seems like a non-starter, and I expect that would break lots of things (I mean it could be done but I'm not sure it's worth it for this). Recording this mostly as a hint than anything else, via another field (if necessary at all) I think would be best. - As covered in previous discussions, dpkg could (but I don't think it's necessary) check whether the .deb is runnable on the current hw, but that's tricky as chrootless installs need to be taken into account, etc. It should certainly not be part of dependency resolution. - I'm not fond of having to change the binary package name format either for this (name_version_arch.deb) even if at least dpkg itself does not care (but I know other tools do care), and depending on the format I'd expect things to break (this goes back to the shared pool concern). - If dpkg-architecture needs to be aware of this, then this might need to be auto-detectable from just the current toolchain being used. Some of the above problems could perhaps be avoided if we introduced a concept of architecture aliases/ISAs (similar to what rpm has), which would side-step the pool sharing issue, the binary package renaming, etc. One big issue with this is that it requires for dpkg to have an exhaustive table of all such aliases, and if there's ever a new alias added, old dpkg versions need to be updated or they will not understand what they match with. So this does not seem ideal either. So I guess this is a variation over your proposal, but perhaps this could still be used in specific contexts, say only at build-time (but not for dependency relationships), for repo management (say binary-arm64v9/Packages.xz), or binary package names where the field would specify the actual name for the filename, say: Architecture: arm64 ArchitectureIsa: arm64v9 or maybe better: Architecture: arm64 ArchitectureIsa: v9 resulting in dpkg-deb generating: binpkg_1.0-1_arm64v9.deb but targeting arm64. I also think I prefer naming this explicitly as ISA variants, if you will, than just architecture variants as that gives way too much room (which perhaps we want, but then that has other implications over compatibility), and for the field perhaps just Isa is better, to avoid the implicit repetition of ArchitectureInstructionSetArchitecture :), but that makes it less easy to associate both as related. In the end though, I think there are perhaps bigger constraints from the infra side of things than the package tooling, stuff like archive management software, or binary transition migration and similar. > In terms of building consensus around this design, I thought it makes sense > to start at the bottom of the stack and so here I am on this mailing list > :-) I guess in due course this could become a DEP, and would certainly need > to be discussed on debian-devel before getting too far. I'm not sure there's ever been much of a wide interest in something like this in Debian TBH. Due to deployment and increased infra overhead at least? > What do you think? Have I missed any glaring implications? No, I think the overall picture is about right, and captures most of the things we have discussed at various times and places in the past. :) > Is there a better way of doing this? I think starting from 5, the rest are probably just details to hammer out, but not insurmountable things. Thanks, Guillem
