On 3/28/2021 16:05, William Hubbs wrote: > On Sat, Mar 27, 2021 at 10:51:11PM -0400, Joshua Kinard wrote: >> On 3/27/2021 20:32, William Hubbs wrote: >>> On Sat, Mar 27, 2021 at 05:43:34PM -0400, Joshua Kinard wrote: >>>> On 3/23/2021 07:31, Rich Freeman wrote: >>>>> On Mon, Mar 22, 2021 at 6:54 PM Andreas K. Huettel <dilfri...@gentoo.org> >>>>> wrote: >>>>>> >>>>>>>> Council decided years ago that we don't support separate /usr without >>>>>>>> an initramfs, but we haven't completed that transition yet. >>>>>>> >>>>>>> Which doesn't imply that we deliberately break things. >>>>>> >>>>>> That's right. Though we should at some point start thinking about an end >>>>>> of support for separate usr without initramfs. >>>>>> >>>>> >>>>> Just to clarify - it is already unsupported at a distro level. It is >>>>> just that some individual packages still work with it. >>>>> >>>>> The current Council decisions on the issue are (just providing for >>>>> general reference): >>>>> >>>>> - "Since that particular setup may already be subtly broken today >>>>> depending on the installed software, Council recommends using an >>>>> early boot mount mechanism, e.g. initramfs, to mount /usr if /usr >>>>> is on a separate partition." >>>>> Accepted unanimously. [1] >>>>> >>>>> - "The intention is to eventually not require maintainers to support >>>>> a separate /usr without an early boot mechanism once the Council >>>>> agrees that the necessary docs/migration path is in place." >>>>> Accepted with 4 yes votes, 1 no vote, 2 abstentions. [1] >>>>> >>>>> - "The Council agrees that all preparations for dropping support for >>>>> separate /usr without an initramfs or similar boot mechanism are >>>>> complete. A news item will be prepared, and users will be given one >>>>> month to switch after the news item has been sent." >>>>> Accepted with 5 yes votes, 1 no vote, 1 abstention. [2] >>>>> >>>>> Current policy documentation: >>>>> Developers are not required to support using separate /usr filesystem >>>>> without an initramfs. [3] >>>>> >>>>> 1 - https://projects.gentoo.org/council/meeting-logs/20130813-summary.txt >>>>> 2 - https://projects.gentoo.org/council/meeting-logs/20130924-summary.txt >>>>> 3 - https://projects.gentoo.org/qa/policy-guide/filesystem.html#pg0202 >>>> >>>> Is there a list of software/ebuilds that currently do this "subtle" >>>> handling >>>> of separate /usr w/o initramfs? >>>> >>>> I've got just my MIPS systems left that use a separate /usr and do not boot >>>> with initramfs because I build fully monolithic kernels and that makes the >>>> resulting vmlinux images run up against hard size limits in the SGI PROM >>>> (firmware, BIOS, etc) on these machines if I try to pack too large of an >>>> initramfs in. I can check for any software that may be switched over soon >>>> to a hard initramfs requirement and look at my options. >>> >>> One group of packages involved in this is any package that calls >>> gen_usr_ldscript. We have this function for a couple of reasons. >>> >>> Gentoo has a policy that bans *.a static libraries from being >>> installed in /lib*. I think this policy is unique to Gentoo, >>> because Most build systems I've seen do not >>> have separate paths for static vs dynamic libraries, so all libraries >>> from a package are installed in /usr/lib* or /lib*. This policy was >>> originally hard coded into portage some time back (I can find the commit >>> if you want it) before it was made a formal policy by the qa team. >>> It has since become a formal policy. >>> >>> Because of this policy, we have to tell upstream build systems to >>> install libraries in ${ED}/usr/$(get_libdir). This is fine unless you >>> have separate usr with no initramfs. In that case, you have binaries on >>> / that link to shared libraries on /usr. When we saw this happening, we >>> started manually moving shared libraries from /usr/lib* to /lib* if >>> someone needed the package at boot time. Then we hit bug 4411 [1] >>> where the linker was prioritizing static libraries in /usr/lib* over shared >>> libraries in /lib*. >>> >>> I don't know if we tried to address that upstream or not, but ultimately >>> gen_usr_ldscript came out of it. >>> >>> Several folks have wanted to get rid of this function for years [2]. >>> >>> I have looked into it before, and there are several things that can be done. >>> >>> We can have packages that currently build static libraries and >>> use gen_usr_ldscript stop building static libraries and use the >>> appropriate setting in their build systems to install libraries in >>> /$(get_libdir). This is the path OpenRC is taking per request of the qa >>> lead. The next release will not support the static-libs use flag. This >>> will actively break anyone who is expecting this use flag to exist for >>> OpenRC. >>> >>> If a package does not build static libraries in the first place, there is >>> really no reason to call gen_usr_ldscript; the ebuild should be using >>> the upstream build system to put the libraries where they need to be. >> >> The number of packages calling gen_usr_ldscript looks to be fairly small. >> Creating a TRACKER bug and sub-bugs for checking or removing the need for >> gen_usr_ldscript might be a way to go to pair the list down and reduce the >> footprint: >> > > Yes, I can start this work. > >> >>> If static libraries are needed for a package that is using >>> gen_usr_ldscript to move shared libraries to /lib*, the only option you >>> have is to drop the gen_usr_ldscript call. If you do that, all of the >>> libraries will stay in /usr/lib*, but as soon as one package takes this >>> path, there will be active breakage for someone who is using a separate >>> /usr without an initramfs. >> >> Correct me if wrong, but static libraries are only needed during >> compilation, right? The *.a files are merged into the binary at link time >> and thus that binary can stand on its own regardless of whether the *.a >> files are available or not. They're not like shared libs which are needed >> by the loader to resolve symbols at run time. > > You are correct, and actually in Gentoo we prefer not to link to static > libraries, so we build them but we do not use them. > > *snip* > >> We're mostly talking about the small window during boot where, on a system >> with /usr on a separate disk partition, /usr might not be available until >> some userspace tool in /bin or /sbin runs to make it available. Running the >> system compiler during boot would be a rare and very unique scenario (not to >> mention a sign of questionable development processes). > > The problem is, there's a chicken-and-egg problem in the scenario where > / and /usr are on separate partitions, and this is why a number of linux > distros have moved to requiring an initramfs in this situation. > I'm linking systemd's description here, only because it is the best > writeup of the issue I've found [1]. > Anything that is needed in the early boot process requires all of its > libraries, > dependent libraries, binaries, data files, etc to be on /, and this has > become a moving target.
Yeah, I've read systemd's explanation, and generally disagree with it. They created the problem in the first place, then invented their own solution for it, and now everyone acts like they're the wise men on the mountain for it. I still don't see the connection to the static *.a libs and whether they're on /lib or /usr/lib, though. Unless we're implying that where the *.a's go, so too do the *.so's go, then THAT makes sense because *.so's ARE needed at program runtime, whereas *.a's are not. I wonder if we couldn't shovel all static libs off to a dedicated folder somewhere, like '/usr/lib/static/<pkg>/*.a', similar to the way debug files are now consolidated under '/usr/lib/debug'. Since they're only needed during a specific kind of compilation that we don't support out-of-the-box that happens long after the system is fully booted, stuffing them off somewhere unimportant would make some sense. Most modern software should be using shared libs by default, and if it ain't, that's either a bug or that software is for a very specific function (like a bootloader). >>> The most controversial thing to do would be the /usr merge. It would >>> affect far more users than the other paths for getting rid of >>> gen_usr_ldscript, but it would make all "separate /usr" concerns go >>> away along with giving us a number of other benefits. >>> >>> The short version is, anything we do to remove the gen_usr_ldscript >>> function will actively break some group of users. >> >> If I read the temperature right, it seems like there is desire to eliminate >> gen_usr_ldscript regardless of sep-usr or not. If so, then that seems the >> way to go. Make the eventual /usr merge a separate issue to tackle some >> time further down the road. > > The way I see it, when we start to remove the gen_usr_ldscript calls, > people using a sep-usr mount without an initramfs will run into one or > both of these issues: > > - they might have to increase the size of their root partition depending > on what gets added to /lib* > - if one package in that list drops gen_usr_ldscript without installing > libraries in /lib*, it will mean they need an initramfs. I tend to make my root partitions ~4GB, which has often been plenty of room for well over 15 years. But again, location of the *.a static libs is irrelevant during system boot. They are not needed nor referenced when a program executes. A statically-compiled program has all of its dependencies lumped inside of it, so you could put it pretty much anywhere on the filesystem and run it (ignoring for a moment 'noexec' potentially being set). Or even more to the point, you could e-mail a static binary to someone else on a different distro and it'll probably execute just fine. The important bit here is certain software packages defaulting their install into /usr that are needed to bring the system up (like encryption, lvm, etc). It's when using those setups that you pretty much have to use an initramfs, sometimes regardless if you have /usr on another partition or not. > From my pov, it is better to start using initramfs in that situation, > and if you do that you fix all of the other issues mentioned on > the page linked above. It feels like the usr merge is less of a big deal > once we know everyone who has /usr separate is using an initramfs. Yeah, and the problem there has been the generation of those initramfs archives. I'm stuck between a rock and a hard place because the classic tools like dracut target popular archs like amd64 or arm64, but not mips, and especially not 20+ year-old hardware, and go on to make assumptions under that pretense. Luckily, my MIPS machine setups are simple and really only rely on mdraid auto-assembling via 0.90 metadata (otherwise, I'd need an initramfs to get the things to boot, cause someone decided at some point that it ain't the kernel's job assemble the array, and newer metadata signatures enforce this...). Add on to that, my specific use-case, I really just need the tiniest binary that tells the kernel to go and mount /usr as an XFS partition on /dev/md2 (after array assembly). No one seems to have cobbled together an initramfs generator that simple. Instead, the Linux world went straight for the Rube Goldberg versions of initramfs. -- Joshua Kinard Gentoo/MIPS ku...@gentoo.org rsa6144/5C63F4E3F5C6C943 2015-04-27 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 "The past tempts us, the present confuses us, the future frightens us. And our lives slip away, moment by moment, lost in that vast, terrible in-between." --Emperor Turhan, Centauri Republic
