Hi, Yeah, I know we have too many /usr-merge discussions. Still, there is reason to continue posting. My last summary/status was https://lists.debian.org/20230712133438.ga935...@subdivi.de from July 12th and I'm giving an update of what happened since then here and explain how I want to move forward.
# DEP17 I've updated the DEP17 MR at https://salsa.debian.org/dep-team/deps/-/merge_requests/5 and continue to provide a rendered version at https://subdivi.de/~helmut/dep17.html. Notable changes: * We learned that in bookworm and beyond, the rootfs of the debian-installer is not /usr-merged. If we start moving files in packages, we'd likely break the installer. (This is DEP17-P10.) The kinda obvious mitigation is performing the merge there (DEP17-M22) and I've submitted a MR to that end. https://salsa.debian.org/installer-team/debian-installer/-/merge_requests/39 * The heated discussion around debootstrap and the bootstrap protocol was tracked down to be rooted in a misunderstanding. The proposal changing debootstrap (DEP17-M19) has gained consensus and has been uploaded to unstable as debootstrap/1.0.130 by Luca Boccassi. Thanks. Please test this version of debootstrap and report bugs as we intend to backport this change and the change merging --variant=buildd in trixie and beyond to bookworm and bullseye. * The issue of loosing empty directories (DEP17-P6) has been clarified to also loose such directories in plain upgrades when they are canonicalized. Therefore all empty directories in aliased locations are affected regardless of whether other packages ship files within. * The issue of loosing empty directories (DEP17-P6) has formalized two known mitigations. M20: Restore empty directories in maintainer scripts M21: Add placeholder files * I have recorded the perceived consensus in a new "Proposal" section. # Archive work * I currently manually file RC bugs for file conflict bugs that affect the /usr-merge. If you also file such bugs, please add the debian...@lists.debian.org usertag "fileconflict" and ensure that you set appropriate affects or assign the bug to multiple packages. * I have filed bugs with patches and MRs for deleting empty directories (DEP17-P6) that I was as unused. After all, a directory that is not installed, cannot be lost. In case of lib32lsan0 and libx32lsan0, this resulted in deletion of the entire package (thanks Matthias). * I have filed bugs for trigger interests on aliased locations (DEP17-P2) and asked maintainers to duplicate them (DEP17-M12). # Continuous monitoring of problems The dumat service introduced earlier continues to detect problems related to /usr-merge. Its detection has been slightly improved and it also associates reported bugs now. If bugs are correctly assigned, usertagged and have the right affects, dumat associates them and displays them in the output file https://subdivi.de/~helmut/dumat.yaml. At the time of this writing 40 of 111 issues are associated with bug reports. MRs are not associated. # Next steps ## Continuous MBF I intend to implement automatic bug filing in dumat for some classes of bugs. The bug association mentioned above has been implemented to avoid filing duplicates. I intend to extend the service to automatically (with no human intervention) file bugs for issues that are not yet associated with a bug number. If you close such a bug without fixing the cause, a new one will be filed. I intend to implement this one category at a time and most of these categories will result in bugs of severity serious. In general, no bugs will be filed for "risky" issues (52 at the time of this writing) and bugs will only be filed for versions in unstable or experimental. In order to limit possible damage, I intend to limit the service to filing one bug per run (i.e. maximum 4 bugs per day). The categories that shall receive automatic rc bugs are: * undeclared file conflicts (all existing ones filed manually) * P1: ineffective replaces (none at present due to the moratorium) * P2: ineffective trigger interest (all existing ones filed manually) * P3: ineffective diversions (none at present due to the moratorium) * P6: loss of empty directories (some filed) * P7: loss of m-a:same shared files (none at present due to the moratorium) The first is usertagged under the qa umbrella: https://udd.debian.org/cgi-bin/bts-usertags.cgi?user=debian-qa%40lists.debian.org&tag=fileconflict The others will be usertagged with my email helm...@debian.org and tag name dep17pN. Why are undeclared file conflicts on this list? They typically are mitigated with Breaks+Replaces or diversions. Both of these mechanisms have potential breakage with aliasing. Therefore, an undeclared file conflict may hide a problem related to /usr-merge. The empty directory loss issues are currently not filed at RC severity even though they tend to be reproducible on bookworm. The moratorium does not prevent these from happening. The first issue of this kind was discovered by Andreas Beckmann using piuparts and for this reason, I intend to upgrade these issues to RC severity. It is easier to deal with them in a well-understood way than in some piuparts report blocking testing migration of another package. In general, the use of RC severity is deliberate in order to block /usr-merge problems from migrating to testing. This approach has been acked by release team member Paul Gevers. I don't currently have bug templates for these and hope that you can trust me with coming up with good templates. Since this is a continuous MBF, I expect to improve templates based on feedback from recipients. Given the current rate of problems introduced, I expect that the service files about 1 bug per week for now. When we lift the moratorium and mass convert packages to canonical locations, I expect it to file more bugs and hope that we do not introduce more than 28 bugs per week, which would exceed the proposed rate limit. I recognize that this is quite a non-standard way to ask for a MBF. Does anyone object to me doing it in this way? Do you need more details on the implementation before I can proceed? In any case, I'm ready to stop the MBF at any point if it causes problems. I will continuously monitor the bugs it files and will be ready to respond as people reply to those bugs. It's not the first time I've raised this MBF, but I go into more detail that previously here. All of the feedback I've got thus far is positive. ## Updating debootstrap Another major piece is updating debootstrap in bookworm and bullseye. There are two changes that Simon McVittie intends to backport: * Merging of --variant=buildd chroots in trixie and beyond. Once this change is deployed to buildds, their chroots will become merged and this is a precondition to lifting the moratorium. * Changing the merge implementation to merge after initial unpack (M19). This change is necessary to be able to ship the aliasing symlinks in a package (M11). It is backwards compatible. ## dh_usrmerge I intend to add a new tool dh_usrmerge to debhelper (not yet implemented). Its purpose is performing the path canonicalization in binary packages. As long as the moratorium is in effect, this helper must not be used. It shall be possible to enable this helper via "Build-Depends: dh-sequence-usrmerge" or "--with=usrmerge". I discussed the possibility of adding this helper to the default sequence via a compatibility level. However, Niels Thykier pointed out that a new compatibility level typically takes 3 years to be adopted and we want 100% adoption before trixie. It will not be mandatory to use this helper. If dropping e.g. --prefix=/ and relying on the /usr default works, that's better. ## Informative MBF I was asked to perform another MBF at minor severity for all (2000) affected packages (collapsed to one per source package) informing maintainers about what's going on. Since path canonicalization can introduce bugs, we want it to happen on an opt-in basis rather than binNMUs and opt-out. In particular, these bug reports will ask affected maintainers to perform their canonicalizing upload to experimental and only proceed to uploading to unstable if they do not receive a bug within three days. In a similar vein, affected maintainers will be asked to upload any change that restructures the package to experimental first (throughout the entire trixie cycle) and also wait three days prior to proceeding to unstable. If these guides are followed, little breakage shall enter unstable. Otherwise, the automatic filing of bugs shall prevent those issues from migrating to trixie. The intention here is to remove the need of having to remember all these crazy problems from the average maintainer's duties and instead have them rely on automatic reporting. I expect that half of the packages can be converted with no changes beyond enabling dh_usrmerge. ## Lifting the moratorium I see the following preconditions for having the moratorium lifted and hope that other CTTE members agree. * buildd chroots need to be /usr-merged. * The debian-installer needs to be /usr-merged. * The automatic RC bug filing service needs to be in operation. * We need to supply tools (e.g. dh_usrmerge) that enable maintainers to do their part with minimal effort and communicate this (informative MBF). ## Mass conversion of packages With the exception of some essential packages, files are moved from / to /usr in binary packages with support from maintainers and in some cases using NMUs. Once most packages are converted, updates to the remaining essential packages including base-files, bash, dash, glibc, and util-linux are prepared and coordinated to be uploaded in a single dinstall with the respective maintainers. When base-files installs the aliasing symlinks, it will provide usr-is-merged. ## Coordinate with downstreams Every derivative that also implements the /usr-merge in this way, will have to monitor their archive in a similar way as we do. An early version of dumat has been run on Ubuntu, but this effectively affects every derivative. We also need to update documentation such as release notes and explain that local packages and addon repositories may be negatively affected. Let me close this mail with asking for feedback. I'm quite sure that this plan will not work out exactly the way it is presented here and it will need refinement. Still it provides a starting point for resolving the issues at hand. If you disagree with any of this or have questions, please speak up. Helmut
