On May 6, 2025, at 08:15, Nuno Teixeira <edua...@freebsd.org> wrote: > Hello Mark, > > Definitely I'm not getting the results I want with WITH_META_MODE using BEs > since most of the times I end up rebuilding almost everything in new BE. > > Should I stop using WITH_META_MODES a go straight to clean builds (clean > /usr/obj) instead?
If by "clean" you mean doing some form of "rm -fr" on all or part of what is by default somewhere under /usr/ob/ : You likely would be deleting files that would be regenerated during a build instead of any of those being reused. May be I've guess wrong about what you mean by "clean /usr/obj"? I'll note that I've never used ccache or the like: https://man.freebsd.org/cgi/man.cgi?query=ccache&sektion=1&n=1 I expect that there are others that have that might comment about such setups. But going in another direction: I do not know why you are doing your own builds instead of using pre-built materials, such as PkgBase's base_latest (updated multiple times per day, not that you need to do updates from it that often) or base_weekly (again, you might not update as often). Do you have local modifications of the system source that you are building? (That could mean not using PkgBase materials, for example.) Just kernel updates? Just world updates? Both? Neither? I also do not know if you build your own packages/ports. So I do not even known if you need a toolchain. Do you need to do package/port builds to get security updates sooner or some such? Can you comment on what has to be true of your system environment(s) for such (even if it in turn means build times that you do not like have to be involved)? Delimiting the tradeoff structure requirements might help in picking a path. A RPi4 specific note: One core on the RPi4B executing code for which the L1 (smallest) level cache is ineffective can saturate its memory subsystem. Depending on details I've had experiments in normal system build activity where using -j3 instead of -j4 or more actually took somewhat less time overall. (Not that the differences were huge or anything. And they might not be systematic. But -j3 can also help if there are also memory usage tradeoffs that need to be managed. Does the RPi4 have 8 GiBytes of RAM? 4 GiByte? 2 GiByte?) A note on official PkgBase build use: The below illustrates that it is possible to mix official PkgBase and personal system builds in some ways. Not that it is on RPi4B's, but I have both PkgBase kernels and my own kernel builds and use /boot/loader.conf to pick which is the default for booting. I do not boot kernels that are older than the boot world on the media. I boot a PkgBase world and have directory trees for chroot or other such use with my personal world builds, some have my personal package builds installed and others have official package builds installed. I only use my personal world builds with my personal kernel builds. (They are matched.) I never use my personal kernel build when it is older than the PkgBase kernel or world that I use. This means my personal kernel build supports booting the PkgBase based world when I use that kernel. Overall I can investigate if any problems I run into are reproducible without my system or package builds involved. I have various poudriere(-devel) jails, some use official PkgBase based jails and others are using my personal builds for the jail content: # poudriere jail -l JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP PATH release-aarch64 14.2-RELEASE-p1 aarch64 pkgbase 2025-05-05 22:11:10 /usr/local/poudriere/jails/release-aarch64 release-armv7 14.2-RELEASE-p2 armv7 pkgbase 2025-03-13 21:50:17 /usr/local/poudriere/jails/release-armv7 official-aarch64 14.2-STABLE aarch64 pkgbase 2025-05-05 22:13:47 /usr/local/poudriere/jails/official-aarch64 official-armv7 14.2-STABLE armv7 pkgbase 2025-03-13 21:47:04 /usr/local/poudriere/jails/official-armv7 main-aarch64 15.0-CURRENT aarch64 pkgbase 2025-05-05 22:15:30 /usr/local/poudriere/jails/main-aarch64 main-CA76 15.0-CURRENT arm64.aarch64 null 2025-02-13 01:35:39 /usr/obj/DESTDIRs/main-CA76-poud main-CA76-bulk_a 15.0-CURRENT arm64.aarch64 null 2025-02-13 01:35:39 /usr/obj/DESTDIRs/main-CA76-poud-bulk_a main-CA7 15.0-CURRENT arm.armv7 null 2025-02-20 18:16:55 /usr/obj/DESTDIRs/main-CA7-poud main-CA7-bulk_a 15.0-CURRENT arm.armv7 null 2025-02-20 18:16:56 /usr/obj/DESTDIRs/main-CA7-poud-bulk_a main-armv7 15.0-CURRENT armv7 pkgbase 2025-03-14 22:48:11 /usr/local/poudriere/jails/main-armv7 # poudriere jail -l JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP PATH release-amd64 14.2-RELEASE-p2 amd64 pkgbase 2025-05-05 22:19:47 /usr/local/poudriere/jails/release-amd64 official-amd64 14.2-STABLE amd64 pkgbase 2025-05-05 22:21:46 /usr/local/poudriere/jails/official-amd64 main-ZNV4 15.0-CURRENT amd64 null 2025-02-12 16:03:46 /usr/obj/DESTDIRs/main-ZNV4-poud main-ZNV4-bulk_a 15.0-CURRENT amd64 null 2025-02-12 16:03:46 /usr/obj/DESTDIRs/main-ZNV4-poud-bulk_a main-ZNV4-dbg 15.0-CURRENT amd64 null 2025-04-02 09:20:08 /usr/obj/DESTDIRs/main-ZNV4-poud-dbg main-amd64 15.0-CURRENT amd64 pkgbase 2025-05-05 22:26:12 /usr/local/poudriere/jails/main-amd64 (I've never bothered with i386 in amd64 contexts and have never used FreeBSD on an actual i386 class system.) > My first concern is to speed up builds expecially on rpi4. Per some prior questions: What other constraints must also be met? > Any hints are welcome. > > Thanks, > > > Mark Millard <mark...@yahoo.com> escreveu (segunda, 5/05/2025 à(s) 23:18): >> Nuno Teixeira <eduardo_at_freebsd.org> wrote on >> Date: Mon, 05 May 2025 20:37:09 UTC : >> >> > (...) >> > >> > Don't forget to `env NO_PKG_UPGRADE=yes beinstall.sh` to not mess with >> > ports and stuff. >> > >> > Nuno Teixeira <edua...@freebsd.org> escreveu (segunda, 5/05/2025 à(s) >> > 21:34): >> > >> > > Hello, >> > > >> > > Using incremental WITH_META_MODE builds, after installation with >> > > beinstall.sh, building same src, a complete compilation happens. >> > > >> > > If someone uses this script, could you please do the following test: >> > > >> > > WITH_META_MODE=yes (/etc/src-env.conf) >> > > filemon module loaded >> > > >> > > # cd /usr/src >> > > # make buildworld-jobs buildkernel-jobs >> >> The above used older commands and files from before >> the following install. META_MODE recorded the use of >> those commands. >> >> Example .meta mode file content: >> >> # Meta data file >> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/libc++.a.meta >> CMD @echo building static c++ library >> CMD @rm -f libc++.a >> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o charconv.o >> chrono.o condition_variable.o condition_variable_destructor.o debug.o >> exception.o filesystem/directory_iterator.o filesyste >> m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o >> ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o >> optional.o random.o random_shuffle.o regex.o shared_mutex.o >> stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o >> utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o >> cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g >> nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o cxxrt_typeinfo.o >> CWD /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++ >> TARGET libc++.a >> -- command output -- >> building static c++ library >> >> -- filemon acquired metadata -- >> # filemon version 5 >> # Target pid 22471 >> # Start 1611359217.214996 >> V 5 >> E 22961 /bin/sh >> R 22961 /etc/libmap.conf >> R 22961 /var/run/ld-elf.so.hints >> R 22961 /lib/libedit.so.7 >> R 22961 /lib/libc.so.7 >> R 22961 /lib/libncursesw.so.9 >> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE >> F 22961 22962 >> E 22962 >> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm >> R 22962 /etc/libmap.conf >> R 22962 /var/run/ld-elf.so.hints >> R 22962 /lib/libc.so.7 >> R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE >> D 22962 libc++.a >> X 22962 0 0 >> . . . >> >> So META_MODE has lots of files that were used >> that it can later detect being newer than the >> prior build results, leading to rebuilds based >> on those newer files. >> >> > > # ./tools/build/beinstall.sh >> >> The new be will have various updated files >> that could be different by content and, for >> commands, could behave differently than those >> used to do the prior build. Detecting newer >> time stamps on such used files leads to >> rebuild activity. >> >> More than /usr/src/ and /usr/obj/ content >> are involved as well. >> >> Note that the new be is based on somewhat >> different files than the original >> buildworld-jobs buildkernel-jobs was based >> on. >> >> > > # reboot >> > > >> >> (I presume booting into the new be here.) >> >> > > # cd /usr/src >> > > # make buildworld-jobs buildkernel-jobs >> >> META_MODE will notice when commands are used >> that are newer than when the prior build was >> done. Similarly for other files that may be >> read. It will make sure that the newer commands >> and files are allowed to produce new results >> that potentially could be distinct in content >> from what the old context produced for results. >> >> > > >> > > Since src and obj are the same from one BE to newer BE, minimal >> > > compilation should happen, not a full one. >> >> META_MODE is more careful than that. >> >> Note: I'm not claiming that new behavior that is >> needed is likely for lots of the files with new >> dates. But META_MODE is biased to avoiding leaving >> in place something that should have been updated. >> >> > > >> > > Am I missing something here? >> > > >> >> Note that make has a -dM option: >> >> M Print debugging information about “meta” mode decisions >> about targets. >> >> So, for example, >> >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cat' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cp' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide' >> is newer than the target... >> file >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/dd' >> is newer than the target... >> . . . >> >> Various . . ./tmp/legacy/. . ./*bin/ actually were >> links to files. >> >> Ultimately buildworld then installworld lead to new >> dates for a bunch of files used. Later use of >> META_MODE notices such and rebuilds based on the >> newer files. (It is a lot of detail to go through >> it all.) >> >> Back in 2021 and 2023 I got help with exploring >> avoiding lots of these. But, in the end, it >> involved use of experimental code in >> share/mk/src.sys.obj.mk to provide a new >> definition to use to build some paths with. >> >> The experiments were an unsupported activity that >> produced an unsupported change to allow >> configurable enabling of taking risks with not >> updating files that possibly should be updated. >> > === Mark Millard marklmi at yahoo.com
