Hi, I fear we are not done with empty directory loss yet. This is a technical update for future reference.
On Wed, May 31, 2023 at 11:59:58AM +0200, Helmut Grohne wrote: > On Tue, May 30, 2023 at 11:53:00AM +0200, Helmut Grohne wrote: > > In effect, this bug report is an instance of a bug class. I am in the > > process of quantifying its effects, but I do not have useful numbers at > > this time. As an initial gauge, I think it is about 2000 binary packages > > that ship empty directories (which does not imply them to be affected, > > rather this is to be seen as a grossly imprecise upper bound). > > I did some more analysis work here and have to admit that I know my data > model has a weakness that may result in false negatives. I'd have to do > a complete reimport of packages and eventually will, so for now I'm > dealing with incomplete data here. I note that content indices do not > cover empty directories, so you really have to download loads of .debs > to find these. We're much further in understanding the problem now. > Anyway, to gauge the problem, we're effectively looking for a > combination of packages A and B such that: > > * A ships an empty directory. > * That empty directory is a path affected by aliasing (either in /usr > or /). > * B also ships that directory (e.g. non-empty) in the "other" > representation of that path. This earlier representation is incomplete. Moving a file from / to /usr (which currently is prohibited by the moratorium) as part of a simple package upgrade also triggers the loss behaviour. In any case, https://subdivi.de/~helmut/dumat.yaml now reports the affected instances (both when multiple packages are involved and upgrade scenarios) and it's not that many. > So yeah, this bug class is clearly not one to panic about. As we move > files from / to /usr, I expect this bug class to gain more occurences. I > am not aware of a generic solution and it seems diversions won't cut it. > If you can propose any generic workaround or recipe for this situation, > I'm all ears. The placeholder file sounds ugly, but might work. We've progressed somewhat since. At this time, my favourite mitigation is deleting affected directories and I've sent a number of patches for applicable situations. Unfortunately, some empty directories exist with reason and we need another mitigation for them. The options seem to be as follows: # M9 While dpkg does not allow diverting a directory, you can trick it into doing that by temporarily moving the directory out of the way. So diversions are an option here though they're really ugly. # M17 A way to not loose these directories is to keep them in both aliased and canonical location. While this works fairly reliably (until we attempt to remove the aliased location), it is incompatible with other mitigations such as shipping the aliasing symlinks in a package (M11). # M20 We can also accept the temporary loss of these directories and restore them using maintainer scripts. In upgrade scenarios, this is trivial as the postinst script runs after the loss event. In the package removal scenario, the package being removed can activate an interest-noawait trigger to restore the directory. I've implemented and attached a prototype for this approach. # M21 The obvious way to avoid loss of empty directories is to make them non-empty. Adding a placeholder file is sufficient here. All of these mitigations can be selected per-occasion (though selecting M17 once precludes a number of other ones). I don't think there is a clear winner here. We can leave the choice of mitigation to the affected maintainers. Rough outline on a per-package basis: * cockpit-tests: deleted #1043322 * firmware-b43-installer: /lib/firmware/b43 triggerless M20 * firmware-b43legacy-installer: /lib/firmware/b43legacy triggerless M20 * fwupd: delete #1041752 * gretl: deleted #1041835 * lib32lsan0 : deleted #1042482 * libjte-dev: deleted #1041753 * libmpeg3-dev: deleted #1041756 * libswe-dev: delete #1041757 * libx32lsan0: deleted #1042482 * netplan-generator: multiple, TBD * openrc: TBD, maybe triggerless M20? * pcp: delete #1041754 * pkg-config/pkgconf/pkgconf-bin: TBD, maybe M21? * printer-driver-foo2zjs: /lib/firmware/hp triggerless M20 * python3-expeyes: deleted #1041755 * systemd,udev: multiple, some deleted, others TBD The takeaway here is that the majority of cases are handled via deletion and we'll be finding solutions with the maintainers of those four packages. If you want to help mitigate these instances, you may add placeholder files or delete empty directories at any time. Adding diversions, maintainer scripts or triggers is something we need to consider at a later stage. Helmut
test.sh
Description: Bourne shell script
