On Sat, 6 May 2023 at 19:51, Helmut Grohne <hel...@subdivi.de> wrote: > > Hi Luca, > > I fear you are still missing too many relevant details. What sounds like > a simple plan is severely broken if carried out in this simple form. It > disappoints me that you continue to present it as this simple given the > earlier evidence. > > On Fri, May 05, 2023 at 11:11:54PM +0100, Luca Boccassi wrote: > > - every package is forcefully canonicalized as soon as trixie is open > > for business > > I gave you scripts to prototype this (via binary patching) and > demonstrated that this immediately makes a fair number of packages rc > buggy due to causing unpack errors. I used zutils as an example. It > really is not that simple and it absolutely cannot be done forcefully, > but must be done in an opt-in way. I won't repeat the rationale here.
Sure, there are some things that need special handling, as you have pointed out. What I meant is that I don't think we need special handling for _all_ affected packages. AFAIK nothing is using diversions for unit files or udev rules, for example (I mean if any package is, please point it out, because I would like a word...). I very strongly suspect this will be a small minority out of the total of ~2k would need this treatment, and the vast majority would not. Do you disagree with this gut feeling? Of course, it goes without saying, we should check this before going forward in any direction. > > - the moratorium on moving files from bin/ sbin/ lib/ _and_ to other > > packages at the same time is maintained from bookworm till trixie, and > > will lifted after trixie ships, and applies implicitly to all the > > ~2000 binary pkgs that are affected by the above > > While the CTTE placed the moratorium until the release of bookworm, the > release team extended it beyond, see > https://lists.debian.org/e1ocdqk-0005ge...@respighi.debian.org. So you > need explicit agreement from the release team on your plan. Failing > that, any package that has been forcefully moved is immediately > rc-buggy due to failing a release team requirement. Of course the release team needs to be on board, no questions about that. But given the idea is to maintain their decision exactly as it stands I wouldn't imagine it would be an issue? Once again, the moratorium is explicitly about moving between locations _and_ packages, in combination, not either/or. From that same email you linked: "Files moving their canonical location between / and /usr (details in [1]) *and* from one binary package to another binary package within one release cycle remain an RC issue unless dpkg supports it." I'm proposing to keep this in place as a general rule, with the new escape hatch that you devised as the only addition. > Again, let me try to fix up your plan: > > 1. Implement a service that reliably notices missing Breaks + Replaces > even in the presence of aliasing. Augment it to notice > canonicalization changes and have it report that Conflicts (or other > measures) are needed in such cases. This should operate for upgrades > from stable to testing, stable to unstable and testing to unstable. There are already distro-wide upgrade piuparts checks run occasionally IIRC, at least I've seen a bug from one being reported this week, so we should be most of the way there already? To be clear, this would be very nice and welcome to have obviously, but I don't think it needs to be a blocker. We don't have such checks for vast parts of Policy, including moving files without Breaks+Replaces as evidenced by the recent MBF, and yet we managed to survive thus far. I don't think it's fair that this workstream should be held to higher standards than the rest of the project. > 2. Add a usr-merge-helper script to usr-is-merged that helps with > instating temporary diversions for the purpose of avoiding file loss. > > 3. Add a dpkg-divert wrapper for duplicating diversions according to > aliasing to usr-is-merged. Add postinst scripts that duplicate > existing diversions. Yep, sounds good > 4. Add canonicalization support to debhelper as an (internal) addon. > Enable this addon in the next compat level. This will again populate > ${misc:Pre-Depends} with "usr-is-merged (>= fixed version)" as needed. > Note that usrmerge's provides is unversioned and doesn't satisfy such > a dependency. As already mentioned, I do not believe this is necessary for _all_ cases. It is necessary for a certain number (that we should ascertain beforehand!) of cases, and we need the machinery implemented for them, but I don't think we should impose this workflow with pre-depends and diversions for all affected packages. I think it should be mandatory for problematic packages such as those you already pointed out, _and_ for cases where the maintainer wants to move files also between binary packages. > 5. Write a policy on how to perform changes and how to move files. Simon > Richter suggested a policy for dpkg-divert already. This needs to be > extended to cover other matters and refined. > > 6. Convert packages one by one by enabling the addon or bumping the > compat level. Such a conversion may require: > * Adding Pre-Depends: ${misc:Pre-Depends} > * Adding Conflicts > * Adding invocations of the usr-merge-helper > * Modifying maintainer scripts to canonicalize diversions > * ... As above. > 7. Continuously monitor the reports from the service and turn that > feedback into patches. > > This plan definitely is incomplete and misses critical steps. I suspect > that Simon Richter could immediately tell at least two flaws if not > more. > > A rather obvious flaw (that Simon Richter already mentioned) is how this > breaks filesystem bootstrap. The policy requirement is that all > essential packages must work in unpacked state provided that they have > been configured at least once. (Please don't ask who added this > requirement to policy. I might be embarrassed.) In reality bootstrap > implementations expect more. In order to perform that initial > configuration, we expect that they work sufficiently well to be able to > run maintainer scripts even in unpacked state prior to initial > configuration. This currently works due to the dynamic linker being > shipped in the non-canonical path it is referenced as. Moving it will > break bootstrap tooling unless we install /lib in some binary package > (as Simon Richter pointed out). However, due to how dpkg handles > directory vs symlink conflicts, it is not clear whether we can > reasonably add it as a symlink to any data.tar without causing havoc. > This definitely needs more research and is one of the areas where > teaching dpkg about aliasing brings significant benefits. Curve ball question: is there anything blocking us from canonicalizing PT_INTERP at the source in gcc so that the essential set (and everything else, really) can work without the lib -> usr/lib symlink? Would anything obvious break? This would be in trixie, so even when unpacked on bookworm for the upgrade case, the loader is guaranteed to be accessible from the canonical path. Heck, we could even consider canonicalizing shebangs in scripts shipped in essential packages? I'd imagine /bin/sh would have the same issue as the loader? > Let me restate that I am still not convinced that we can pull this stunt > without painting us in a very dark corner. Too many moving parts of this > approach do not have well-understood solutions. And even then, my > analysis has mainly focused on packages shipped by Debian. It neglected > other relevant aspects such as: > * custom Debian packages (e.g. equivs, config-package-dev, private > packages) > * vendor packages > * local diversions > * local statoverrides > * probably more I don't think we should lose sleep over third party/proprietary packages. Given most of those are autogenerated from dubious scripts (I have seen things...) I doubt they are using diverts and such, most packages I see from companies are built using cmake or java built-in scripts that shove stuff in /opt and manually generate the .ar and pack them, so... I mean, we were not concerned at all when Canonical switched to zstd for deb compression, creating a massive ticking time bomb given the vast majority of third party .debs are built on some flavour of Ubuntu and ship a single .deb for all consumers, and thus would stop being installable on Debian (because for years the dpkg patch was rejected, thankfully now it's all sorted and we are good for bookworm), I don't think we should spend time trying to pre-empt fixing those. Noting in the release notes that in case they do X and Y and Z they will need adjustments and pointing to appropriate documentation seems more than enough to me, no? The most popular ones (by virtue of the very fair and impartial property of being currently installed on my laptop) like chrome, vscode, edge, steam and signal do not ship anything in the legacy directories anyway. Same for local modifications, documenting in the release notes what corner cases people need to be aware of and how to deal with them seems standard practice to me when doing new releases. For example there's a bunch of things you were supposed to do on your machine when upgrading to Bullseye as documented at: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#upgrade-specific-issues and without checking, I'm pretty sure this is normal for all releases. Kind regards, Luca Boccassi
