On May 6, 2025, at 10:47, Nuno Teixeira <edua...@freebsd.org> wrote:
> I use CURRENT on my laptop and rpi4 for some time now.
> I like to see changes in src tree and then rebuild world and test for myself.
> One of the things that attracts me is building and rebuilding stuff and maybe
> that's why I'm so happy as a ports committer :)
Ahh. Okay.
This may get into how much risk of broken builds
in non-obvious ways that you can tolerate. Also:
How much context analysis and tracking you want
to do vs. not.
"man 7 build" references the likes of NO_CLEAN
and NO_CLEANDIR and NO_KERNCLEAN that migth be
used without use of META_MODE. That does not
notice as much evdience of needing rebuild
activity.
NO_CLEAN If set, no object tree files are cleaned at all.
This is the default when WITH_META_MODE is used
with filemon(4) loaded. See src.conf(5) for
more details. Setting NO_CLEAN implies
NO_KERNELCLEAN, so when NO_CLEAN is set no ker-
nel objects are cleaned either.
There is also:
KERNFAST If set, the build target buildkernel defaults to
setting NO_KERNELCLEAN, NO_KERNELCONFIG, and
NO_KERNELOBJ. When set to a value other than 1 then
KERNCONF is set to the value of KERNFAST.
and:
WORLDFAST If set, the build target buildworld defaults to
setting NO_CLEAN, NO_OBJWALK, and will skip most
bootstrap phases. It will only bootstrap li-
braries and build all of userland. This option
should be used only when it is known that none
of the bootstrap needs changed and that no new
directories have been connected to the build.
These sorts of things get into your tracking context
to know when to use them vs. not: more manual
management. When things go wrong, they get into
figuring out how to fix them --possibly involving
bigger rebuilds than otherwise at that point.
> The only "change" that I do in build world is:
> /etc/src.conf:
> WITH_MALLOC_PRODUCTION=yes
> WITHOUT_LLVM_ASSERTIONS=yes
> that might be incompatible with pkgbase since those settings are for releases.
PkgBase has both Debug and non-Debug kernels for
main. It has builds of main and stable/14 as well,
not just 14.* releases.
Just FYI . . .
PkgBase has available (for amd64 and aarch64 and . . .):
main: base_latest/ and base_weekly/
Note: both Debug and non-Debug kernels are available
for main. To my knowledge, PkgBase is the first form
of official distribution of non-debug main kernels.
(I have both types installed.)
stable/14: base_latest/ and base_weekly/
releng/14.*: base_release_[3210]/
( Also: kmods_latest_2/ kmods_quarterly_2/ with some
kmods )
PkgBase has debug symbol file packages available
to install and has its own /usr/src/ and /usr/src/sys/
packages (no git repo).
PkgBase is (always?) based on WITH_LLVM_ASSERTIONS as
far as I know.
However, I'm not aware of PkgBase yet publishing
anything listing with WITH_* and WITHOUT_* settings
that are used for any of its builds --or other such
details. I learned about assert by hitting an example
failure. The status could have changed.
I do not know the WITH*_MALLOC_PRODUCTION status.
Do you really want/need the RPi4B to be able to build
for the powerpc* and amd64 and the like? Might src.conf
be used for:
WITH_LLVM_TARGET_AARCH64=
WITH_LLVM_TARGET_ARM=
WITHOUT_LLVM_TARGET_MIPS=
WITHOUT_LLVM_TARGET_POWERPC=
WITHOUT_LLVM_TARGET_RISCV=
WITHOUT_LLVM_TARGET_X86=
Such could save some time on a RPi4B. A similar point
might go for the laptop (enabling X86 instead?).
WITHOUT_LLVM_TARGET_ALL= might be an option but it used
to be there as an odd build consequence that I ran into
that was tied to if the bootstrap toolchain build could
be avoided vs. not. I've not retested and stuck with
being explicit.
Looks like there is also now: WITH*_LLVM_TARGET_BPF
but it seems to be WITHOUT_LLVM_TARGET_BPF by default.
WITHOUT_LLVM_TARGET_MIPS also looks to be the default
for MIPS these days [on main].
Another thing you may not be interested in using is the
results of WITH_CLANG_FULL:
WITHOUT_CLANG_FULL
Avoid building the ARCMigrate, Rewriter and StaticAnalyzer
components of the Clang C/C++ compiler.
Again, possibly avoiding the RPi4B spending the time
building what is not used.
> Cheers,
>
> Mark Millard <mark...@yahoo.com> escreveu (terça, 6/05/2025 à(s) 17:52):
> On May 6, 2025, at 08:15, Nuno Teixeira <edua...@freebsd.org> wrote:
>
> > Hello Mark,
> >
> > Definitely I'm not getting the results I want with WITH_META_MODE using BEs
> > since most of the times I end up rebuilding almost everything in new BE.
> >
> > Should I stop using WITH_META_MODES a go straight to clean builds (clean
> > /usr/obj) instead?
>
> If by "clean" you mean doing some form of "rm -fr" on all
> or part of what is by default somewhere under /usr/ob/ :
> You likely would be deleting files that would be
> regenerated during a build instead of any of those being
> reused.
>
> May be I've guess wrong about what you mean by "clean
> /usr/obj"?
You did not comment about the above.
WITH_CLEAN is an example of deleting things that might
otherwise be reused instead of being built. I should
have mentioned that above.
> I'll note that I've never used ccache or the like:
>
> https://man.freebsd.org/cgi/man.cgi?query=ccache&sektion=1&n=1
>
> I expect that there are others that have that might
> comment about such setups.
>
> But going in another direction: I do not know why you are
> doing your own builds instead of using pre-built materials,
> such as PkgBase's base_latest (updated multiple times per
> day, not that you need to do updates from it that often)
> or base_weekly (again, you might not update as often).
>
> Do you have local modifications of the system source that
> you are building? (That could mean not using PkgBase
> materials, for example.) Just kernel updates? Just
> world updates? Both? Neither?
>
> I also do not know if you build your own packages/ports. So
> I do not even known if you need a toolchain. Do you need
> to do package/port builds to get security updates sooner
> or some such?
>
> Can you comment on what has to be true of your system
> environment(s) for such (even if it in turn means build
> times that you do not like have to be involved)?
> Delimiting the tradeoff structure requirements might help
> in picking a path.
>
>
> A RPi4 specific note:
>
> One core on the RPi4B executing code for which the L1
> (smallest) level cache is ineffective can saturate its
> memory subsystem. Depending on details I've had
> experiments in normal system build activity where using
> -j3 instead of -j4 or more actually took somewhat less
> time overall. (Not that the differences were huge or
> anything. And they might not be systematic. But -j3 can
> also help if there are also memory usage tradeoffs that
> need to be managed. Does the RPi4 have 8 GiBytes of RAM?
> 4 GiByte? 2 GiByte?)
>
>
> A note on official PkgBase build use:
>
> The below illustrates that it is possible to mix
> official PkgBase and personal system builds in some
> ways.
>
> Not that it is on RPi4B's, but I have both PkgBase
> kernels and my own kernel builds and use
> /boot/loader.conf to pick which is the default
> for booting. I do not boot kernels that are
> older than the boot world on the media.
>
> I boot a PkgBase world and have directory trees for
> chroot or other such use with my personal world
> builds, some have my personal package builds
> installed and others have official package builds
> installed.
>
> I only use my personal world builds with my personal
> kernel builds. (They are matched.) I never use my
> personal kernel build when it is older than the
> PkgBase kernel or world that I use. This means
> my personal kernel build supports booting the
> PkgBase based world when I use that kernel.
>
> Overall I can investigate if any problems I run into
> are reproducible without my system or package builds
> involved.
>
> I have various poudriere(-devel) jails, some use
> official PkgBase based jails and others are using my
> personal builds for the jail content:
>
> # poudriere jail -l
> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP
> PATH
> release-aarch64 14.2-RELEASE-p1 aarch64 pkgbase 2025-05-05
> 22:11:10 /usr/local/poudriere/jails/release-aarch64
> release-armv7 14.2-RELEASE-p2 armv7 pkgbase 2025-03-13
> 21:50:17 /usr/local/poudriere/jails/release-armv7
> official-aarch64 14.2-STABLE aarch64 pkgbase 2025-05-05
> 22:13:47 /usr/local/poudriere/jails/official-aarch64
> official-armv7 14.2-STABLE armv7 pkgbase 2025-03-13
> 21:47:04 /usr/local/poudriere/jails/official-armv7
> main-aarch64 15.0-CURRENT aarch64 pkgbase 2025-05-05
> 22:15:30 /usr/local/poudriere/jails/main-aarch64
> main-CA76 15.0-CURRENT arm64.aarch64 null 2025-02-13
> 01:35:39 /usr/obj/DESTDIRs/main-CA76-poud
> main-CA76-bulk_a 15.0-CURRENT arm64.aarch64 null 2025-02-13
> 01:35:39 /usr/obj/DESTDIRs/main-CA76-poud-bulk_a
> main-CA7 15.0-CURRENT arm.armv7 null 2025-02-20
> 18:16:55 /usr/obj/DESTDIRs/main-CA7-poud
> main-CA7-bulk_a 15.0-CURRENT arm.armv7 null 2025-02-20
> 18:16:56 /usr/obj/DESTDIRs/main-CA7-poud-bulk_a
> main-armv7 15.0-CURRENT armv7 pkgbase 2025-03-14
> 22:48:11 /usr/local/poudriere/jails/main-armv7
>
> # poudriere jail -l
> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP
> PATH
> release-amd64 14.2-RELEASE-p2 amd64 pkgbase 2025-05-05 22:19:47
> /usr/local/poudriere/jails/release-amd64
> official-amd64 14.2-STABLE amd64 pkgbase 2025-05-05 22:21:46
> /usr/local/poudriere/jails/official-amd64
> main-ZNV4 15.0-CURRENT amd64 null 2025-02-12 16:03:46
> /usr/obj/DESTDIRs/main-ZNV4-poud
> main-ZNV4-bulk_a 15.0-CURRENT amd64 null 2025-02-12 16:03:46
> /usr/obj/DESTDIRs/main-ZNV4-poud-bulk_a
> main-ZNV4-dbg 15.0-CURRENT amd64 null 2025-04-02 09:20:08
> /usr/obj/DESTDIRs/main-ZNV4-poud-dbg
> main-amd64 15.0-CURRENT amd64 pkgbase 2025-05-05 22:26:12
> /usr/local/poudriere/jails/main-amd64
>
> (I've never bothered with i386 in amd64 contexts and
> have never used FreeBSD on an actual i386 class system.)
>
> > My first concern is to speed up builds expecially on rpi4.
>
> Per some prior questions: What other constraints must
> also be met?
>
> > Any hints are welcome.
> >
> > Thanks,
> >
> >
> > Mark Millard <mark...@yahoo.com> escreveu (segunda, 5/05/2025 à(s) 23:18):
> >> Nuno Teixeira <eduardo_at_freebsd.org> wrote on
> >> Date: Mon, 05 May 2025 20:37:09 UTC :
> >>
> >> > (...)
> >> >
> >> > Don't forget to `env NO_PKG_UPGRADE=yes beinstall.sh` to not mess with
> >> > ports and stuff.
> >> >
> >> > Nuno Teixeira <edua...@freebsd.org> escreveu (segunda, 5/05/2025 à(s)
> >> > 21:34):
> >> >
> >> > > Hello,
> >> > >
> >> > > Using incremental WITH_META_MODE builds, after installation with
> >> > > beinstall.sh, building same src, a complete compilation happens.
> >> > >
> >> > > If someone uses this script, could you please do the following test:
> >> > >
> >> > > WITH_META_MODE=yes (/etc/src-env.conf)
> >> > > filemon module loaded
> >> > >
> >> > > # cd /usr/src
> >> > > # make buildworld-jobs buildkernel-jobs
> >>
> >> The above used older commands and files from before
> >> the following install. META_MODE recorded the use of
> >> those commands.
> >>
> >> Example .meta mode file content:
> >>
> >> # Meta data file
> >> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/libc++.a.meta
> >> CMD @echo building static c++ library
> >> CMD @rm -f libc++.a
> >> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o
> >> charconv.o chrono.o condition_variable.o condition_variable_destructor.o
> >> debug.o exception.o filesystem/directory_iterator.o filesyste
> >> m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o
> >> ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o
> >> optional.o random.o random_shuffle.o regex.o shared_mutex.o
> >> stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o
> >> utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o
> >> cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g
> >> nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o cxxrt_typeinfo.o
> >> CWD /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++
> >> TARGET libc++.a
> >> -- command output --
> >> building static c++ library
> >>
> >> -- filemon acquired metadata --
> >> # filemon version 5
> >> # Target pid 22471
> >> # Start 1611359217.214996
> >> V 5
> >> E 22961 /bin/sh
> >> R 22961 /etc/libmap.conf
> >> R 22961 /var/run/ld-elf.so.hints
> >> R 22961 /lib/libedit.so.7
> >> R 22961 /lib/libc.so.7
> >> R 22961 /lib/libncursesw.so.9
> >> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE
> >> F 22961 22962
> >> E 22962
> >> /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm
> >> R 22962 /etc/libmap.conf
> >> R 22962 /var/run/ld-elf.so.hints
> >> R 22962 /lib/libc.so.7
> >> R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE
> >> D 22962 libc++.a
> >> X 22962 0 0
> >> . . .
> >>
> >> So META_MODE has lots of files that were used
> >> that it can later detect being newer than the
> >> prior build results, leading to rebuilds based
> >> on those newer files.
> >>
> >> > > # ./tools/build/beinstall.sh
> >>
> >> The new be will have various updated files
> >> that could be different by content and, for
> >> commands, could behave differently than those
> >> used to do the prior build. Detecting newer
> >> time stamps on such used files leads to
> >> rebuild activity.
> >>
> >> More than /usr/src/ and /usr/obj/ content
> >> are involved as well.
> >>
> >> Note that the new be is based on somewhat
> >> different files than the original
> >> buildworld-jobs buildkernel-jobs was based
> >> on.
> >>
> >> > > # reboot
> >> > >
> >>
> >> (I presume booting into the new be here.)
> >>
> >> > > # cd /usr/src
> >> > > # make buildworld-jobs buildkernel-jobs
> >>
> >> META_MODE will notice when commands are used
> >> that are newer than when the prior build was
> >> done. Similarly for other files that may be
> >> read. It will make sure that the newer commands
> >> and files are allowed to produce new results
> >> that potentially could be distinct in content
> >> from what the old context produced for results.
> >>
> >> > >
> >> > > Since src and obj are the same from one BE to newer BE, minimal
> >> > > compilation should happen, not a full one.
> >>
> >> META_MODE is more careful than that.
> >>
> >> Note: I'm not claiming that new behavior that is
> >> needed is likely for lots of the files with new
> >> dates. But META_MODE is biased to avoiding leaving
> >> in place something that should have been updated.
> >>
> >> > >
> >> > > Am I missing something here?
> >> > >
> >>
> >> Note that make has a -dM option:
> >>
> >> M Print debugging information about “meta” mode
> >> decisions
> >> about targets.
> >>
> >> So, for example,
> >>
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk'
> >> is newer than the target...
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb'
> >> is newer than the target...
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cat'
> >> is newer than the target...
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cp'
> >> is newer than the target...
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen'
> >> is newer than the target...
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide'
> >> is newer than the target...
> >> file
> >> '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/dd'
> >> is newer than the target...
> >> . . .
> >>
> >> Various . . ./tmp/legacy/. . ./*bin/ actually were
> >> links to files.
> >>
> >> Ultimately buildworld then installworld lead to new
> >> dates for a bunch of files used. Later use of
> >> META_MODE notices such and rebuilds based on the
> >> newer files. (It is a lot of detail to go through
> >> it all.)
> >>
> >> Back in 2021 and 2023 I got help with exploring
> >> avoiding lots of these. But, in the end, it
> >> involved use of experimental code in
> >> share/mk/src.sys.obj.mk to provide a new
> >> definition to use to build some paths with.
> >>
> >> The experiments were an unsupported activity that
> >> produced an unsupported change to allow
> >> configurable enabling of taking risks with not
> >> updating files that possibly should be updated.
> >>
> >
===
Mark Millard
marklmi at yahoo.com
