Alastair Hogge wrote in <[email protected]>: > The biggest problem I had come across, was the "tags=package={foo}" > declarations in ${SRC}/etc/mtree/*.dist. I had to remove a couple of > these from a 2 or 3 mtree dists. What is going on in this part of the > build infrastructure? There is a history of similar commits[1][2][3], to > the changes I now have to keep locally, yet, des@ recently added[4] more > tags. short answer: as of last week, des's commit is correct and the previous commits removing the tags should be reverted, i just didn't get around to doing it yet.
the problem is that previously, tags=package=foo would cause the 'foo' package to be unconditionally created, even if it was otherwise empty. this first came to light when i introduced several of these in commit 436618a427b4[0], which created several of these empty packages in specific circumstances i hadn't tested (mostly involving non-amd64 builds and certain src.conf options). [0] https://cgit.freebsd.org/src/commit/?id=436618a427b4baaf42d8221ef07d14e3ba787d3a "etc/mtree: Add package tags for /usr/include" in at least a couple of cases this broke the build or install because we'd end up with a 'foo-dev' package having a depend on a 'foo' package that doesn't exist. the post-436618a427b4 reverts are to fix specific instances of this without having to revert the entire initial commit which was broadly correct. this was on my to-do list for a long time until it came up again in D55408[1], the review for des's commit that you mentioned, at which point i fixed the problem properly in 7965c93e4d41[2] by not creating packages which only contain directories. this also fixes all the previous examples of this problem, which is why the workarounds can now be reverted. [1] https://reviews.freebsd.org/D55408 "build: Move all of lp under LPR option" [2] https://cgit.freebsd.org/src/commit/?id=7965c93e4d4103ba6ed7ac1e5f1599c93cbcdbf7 "packages: Don't create empty packages" if you're on stable/15 (or 15.0), i didn't yet MFC 7965c93e4d41, but it's fairly self-contained and should be fine to cherry-pick. if you're rather wait, it will land in 15.1. > From what I understand when investigating this, the build infra uses > both the mtree dists, and ${SRC}/release/packages/generate-set-ucl.lua > to generate the pkgbase sets? Is that correct? Does that mean there is > duplicated work here, and one of these paths is redundant? pkgbase *sets* are generated purely based on annotations (a type of metadata) in the packages we previously built, i.e., we do the full pkgbase build, then scan that for 'set' annotations and use that to generate the set packages. this is driven by release/packages/create-sets.sh, which invokes generate-set-ucl.lua as part of its work. but note that this is only about set packages (i.e., "FreeBSD-set-*"), which never contain any files or directories and should never be referenced in mtree. mtree is involved in the staging process we do prior to the initial (non-set) pkgbase build. that process is driven by Makefile.inc1, and looks roughly like this: * a 'make stageworld' is done to populate <objdir>/worldstage. stageworld is essentially the same thing as installworld, with some minor tweaks to prevent the host system state from affecting the installed world. part of this process involves running mtree to create the directory structure. this stageworld is done with NO_ROOT and METALOG set, which means a complete log of everything installed (including by mtree) is written to a special file called METALOG in the worldstage/ directory. * then we run release/scripts/mtree-to-plist.awk on the METALOG file to generate the individual *.plist files for each package. this looks at the 'package' tag for each file or directory in the METALOG to determine which package the file belongs to. despite the name, this script is never run on mtree files from src. it's called mtree-to-plist because the METALOG is in mtree format. * based on the plist files, we run release/packages/generate-ucl.sh to generate a UCL file containing pkg-create(8) metadata for each package. * we build the packages and copy them to ${REPODIR}. i find this process both overcomplicated and somewhat fragile, so i'd like to improve it at some point, but there are more urgent things to fix like release media builds and some kernel-related issues (e.g. D54542[4] and D54282[5], if you're interested). [4] https://reviews.freebsd.org/D54542 "release: Build the release media from packages" [5] https://reviews.freebsd.org/D54282 "packages: Always install kernel as /boot/kernel.NAME" > I also noted that a few WITHOUT_foo options had no effect on the World > building stage 1.2, bootstrap tools. Here, when WITHOUT_KERBEROS is > defined, it is still built. I even still have a Kerberos KDC pkgbase > component. i am not sure what's going on here since i'm not very familiar with how the src bootstrap works. if no one else has an idea, you could ask on hackers@ (or in a new thread, since people might ignore pkgbase threads.)
signature.asc
Description: PGP signature
