Hi,
this installment serves a dual purpose. Let me first give an update of
the status quo and then pose a consensus question on how we want to deal
with a particular problem.
I Cc d-release@l.d.o as upgrades are an integral part of releases.
I Cc d-ctte@l.d.o for advisory feedback with experience due to earlier
decisions on merged-/usr.
# Status
As I detailed earlier, diversions have been proving more difficult than
anticipated. I spent significant time on molly-guard to get to a working
mitigation and thanks to Francois and Daniel, all of the diverters of
/sbin/halt and others have been updated in experimental for wider
testing. This is looking promising and passing all testing that has been
performed thus far.
Meanwhile Chris Hofstaedler and kind folks in Cambridge worked a lot on
M-A:same shared file loss (DEP17 P7) and got us down to one
(reintroduced) issue. Pending further reintroductions, this aspect is
done. Cool! I've since uploaded debhelper and dh_installudev will now
also install to /usr. udevdir in udev.pc has been changed in a NEW
upload to experimental as well and is expected to hit unstable before
too long (thanks Michael and Luca).
Earlier, I requested a pause of /usr merges. Since we have a better
understanding and solutions that seem to be working now, I am happy for
you to move stuff again more widely. For moves involving diversions in
any way, consider having me review your change ahead of upload.
At the time of this writing, there are 1237 source packages in unstable
that still ship something aliased. This is the number we need to get
down to 0 for trixie. Of these 860 involve a systemd unit and of these
761 only have systemd units aliased many of which can be converted by a
no-change upload due to changed debhelper and systemd.pc behaviour.
# The problem with conflicts
The idea in DEP17 was to use Conflicts as a mitigation strategy in
agreement with a naive reading of Debian policy. As it turns out, that
doesn't exactly match reality (#1057199 debian-policy) and there are
situations where files can be lost despite Conflicts having been
declared. In theory, this subtlety should be irrelevant and
unobservable, but aliasing (which breaks dpkg's assumptions) makes this
observable.
We move a file from / to /usr in $PKGA.
AND one of
The file is also being moved to a different package (causing DEP17 P1).
OR
The file is being diverted (causing DEP17 P3).
AND
The mitigation involves declaring a Conflict for unpack ordering (i.e.
M7 for P1 or M18 for P3).
AND one of
The upgrade is being performed using a direct dpkg invocation
without apt in a way that unpacks the package declaring the conflict
before the conflicted package is removed. Example: #1058937 (Ben's
libnfsidmap1 bug)
OR
The involved packages declare a mutual conflict (or mutual conflicts
+ breaks) and therefore apt invokes dpkg as in the earlier point.
Example: An earlier version of the molly-guard mitigation declared
versioned Breaks for systemd-sysv.
This condition is complex, so let me try to break it down into something
simpler. We'll have somewhere between 20 and 100 instances of P1 + P3 I
guess and we aimed for mitigating most of them using Conflicts (i.e.
first two conditions). The horny part is the last one. It basically says
that as long as we only ever use apt and avoid mutual conflicts, the
issue is not practically observable.
That mutual conflict condition is delicate on its own. There are
basically two ways to trigger it. The way my molly-guard patch did it
was having two versioned Conflicts or Breaks declarations. I checked the
archive and there is no instance of any package combination doing this.
Hypothetically, another way to trigger this is unversioned Conflicts
combined with a package that drops Provides in a later version (thanks
David), but we haven't seen any practical instance and I haven't figured
a good way to gauge this problem yet.
## Options (combinations possible)
When mitigating P1, we can opt for protective diversions (M8) instead of
Conflicts (M7), though that is more fragile.
When mitigating P3, we can avoid the mutual conflicts. For molly-guard
that has been more involved, but it seems manageable. For other
packages (that do not need to access diverted files), it becomes
simpler.
We can restore lost files in a postinst. For this to work, we must
duplicate (e.g. hard link) affected files in the data.tar.
Example: #1057220 (systemd-sysv upgrade file loss)
Note that this approach is not policy compliant for essential packages
as they must work when unpacked and this is relevant for gzip being
diverted by zutils for instance.
We can introduce "barrier" packages (one or more) and have them enforce
conflicting packages removed before the conflictor being unpacked
(thanks Julian).
We can - and this is the crux of the matter - argue that upgrading with
bare dpkg is unsupported and you get to keep the pieces if you do so
anyway.
## Upgrading using dpkg directly?
We already have quite a number of packages that use Conflicts to prevent
file loss in upgrades in a very similar way to #1058937 (Ben's
libnfsidmap1 bug) even in released versions of Debian. For instance,
dhcpcd-base's Replaces were upgraded to Conflicts, see #1053657. If you
employ dpkg, you can still experience the problem there.
Is it ok to call upgrade scenarios failures that cannot be reproduced
using apt unsupported until we no longer deal with aliasing?
If the answer is yes here, we'll close #1058937 (Ben's libnfsidmap1 bug)
with no action calling the scenario unsupported.
If the answer is no here, we'll probably look into adding barrier
packages to multiple suites (including stable releases) and increasing
the number of protective diversions (M8). The question becomes: Is the
cure really better than the disease?
Helmut
