Hi, I'm doing yet another /usr-merge thread even though we already have too many, I know.
The first one was about general discussion and problem analysis. In that first thread, I posted a number of scripts for analyzing problems and snapshot analysis data. In that second thread, we tried to gather consensus around some of the views expressed. # Continuous monitoring for problems This thread hopefully becomes more of a FYI than a discussion. I've turned those hacky scripts into some Python code that continuously (4 times a day) analyzes the archive for some of the problems summarized in DEP17. Interested parties may find the code at https://salsa.debian.org/helmutg/dumat. The results of the analysis[1] (around 2000 lines) are updated at https://subdivi.de/~helmut/dumat.yaml. While it says "issue" there, most of them are *not* actionable right now. Please don't panic. Keep in mind that the moratorium is still active (and that it prevents any of the "risky" ones from becoming practically relevant). There is one kind of issue that may be actionable right now and that's the class of "empty directory" issues. If you notice that such an empty directory actually is not necessary (which probably is the case in the majority of cases), please go ahead and delete it from your package[2] or a file a bug with a patch. Also, please declare Conflicts or Replaces as usual as some forms of undeclared file conflicts also show up in the analysis. Another thing that helps now is cleaning up really old and unversioned Replaces as those may result in false negative detections. What does dumat not detect? * DEP17-P4: Disagreeing alternatives are not a problem as long as we don't canonicalize alternatives (DEP17-M13). I think we can defer this without causing extra pain. * DEP17-P5: dpkg-statoverrides not matching the files shipped. Possibly, I can extend dumat to cover unconditional statoverrides. * DEP17-P8: Filesystem bootstrap. The test matrix is really small, so we'll probably notice when it gets broken. * DEP17-P9: Loss of aliasing symlinks. We can reliably address this centrally e.g. via DEP17-M4. # A rough outlook Let me give a rough idea on how I would like to move forward with this and hope you agree with it. For one thing, we need an agreement on the mitigations that we apply. Except for the bootstrapping aspect, this seems relatively clear. That discussion will likely continue and conclude eventually. ## A proposed process Some of the mitigations are non-trivial to implement and cannot be done e.g. by the janitor, but the dumat.yaml file tells us that the number of occasions where we need them will be fairly low. It's the exceptional case that goes wrong badly. Since the matter is fairly complex and since breakage is rare, I would like to move forward in a way where we do not ask maintainers to pay attention to /usr-merge problems proactively, but reactively. That works with two mechanisms: * We generally ask maintainers to upload some classes of changes to experimental first. Those include: + Moving files from / to /usr. + Moving files between packages. + Changing diversions. + Changing path-based trigger interest. + When in doubt. * You allow me to turn that dumat.yaml tool into an automatic rc bug filing service. The big benefit of this approach is that it lifts the mental load of the matter from individual maintainer's brains. You have a simple rule (use experimental when in doubt) and if your change poses any issue, you receive an rc bug (for that experimental package if you uploaded there or possibly for sid where it acts as migration blocker). My expectation is that such bugs are rare events. If you don't receive an rc bug within two days, assume that your change is fine. I am aware that having an automatic bug filing service with no human supervision (ahead of filing) is something we never had before. Much less for rc bugs. Before enabling this, you definitely want to see a bug template. Are there general objections to this idea? I already talked to Paul Gevers and he sees this as the preferred interface to implement a migration blocker. ## Usertagging bugs In order to avoid filing duplicates, I need a usertagging scheme for bugs. Are there opinions on what user I should use for this? In the simplest instance, I can use my DD login. Roughly speaking every issue type shall translate to an individual usertag. Is there a common usertag for undeclared file conflicts to reuse? # Other aspects ## Informative MBF When I discussed this in Hamburg, there was a request to actively reach out to affected maintainers by sending a minor bug for every source package that is somehow involved. The bug would ask maintainers to move their files from / to /usr, how to do so, the need to do this via experimental first and the chances of getting an rc bug in the process. Do people who have not been to Hamburg Reunion 2023 confirm this? ## No good solution for bookworm-backports There is one major issue where I don't have a good answer: bookworm-backports. When this originally surfaced, Luca Boccassi suggested that we do not canonicalize in backports. That's easier said than done as the support for split-/usr will soon vanish from packages. Thanks to Simon Richter for pointing at this missing piece of the puzzle recently. If we were to canonicalize in bookworm-backports (as needed), we'd need much the same infrastructure as in trixie (e.g. the usrmerge-support package). Adding such intrusive changes to bookworm-backports and pulled by a significant fraction of backports sounds bad to me. The alternative here is that backporting will become a lot harder as those performing backports would have to undo the canonicalization. Worse, such backports pose new upgrade paths that may result in more mitigations in unstable to support bookworm-backports -> trixie upgrades. To make matters worse, an upload to bookworm-backports is immediately available to users and there is no migration that some check (such as dumat) could hook into. What I could easily offer is extending dumat such that you can analyze the problems that would result from uploading a given .changes file, but it only tells about a subset of problems. I wish I was able to tell a better story for bookworm-backports and welcome ideas. ## Another DEP17-P6 mitigation Please allow me to biggy-back one more mitigation onto this mail. The problem of loosing empty directories (DEP17-P6, originally reported by Andreas Beckmann) can also be addressed by shipping the directory in *both* aliased and canonical location. This technically is a violation of current Debian policy and this also is at least partially incompatible with DEP17-M4 (shipping aliasing links in a package). This technique is currently employed by gcc-snapshot (not sure if accidentally). I shall be adding this to DEP17 even though I think we should not employ it. Let me know if you think otherwise. I see it got way longer than intended, but at the same time all of the topics raised seem relevant to me. Hope we get more agreement this time. Helmut [1] If you are really deeply interested and want to redo the analysis without downloading 300GB of Debian packages, you may also download https://subdivi.de/~helmut/dumat.sql.zst to perform the actual analysis yourself. It runs in less than a minute on a beefy machine and this is really where the interesting bits happen. [2] List of packages that *may* be actionable: fwupd, gcc-snapshot, gretl, lib32lsan0, libjte-dev, libmpeg3-dev, libswe-dev, libx32lsan0, pcp, pkg-config, pkgconf, pkgconf-bin, python3-expeyes, systemd
