Re: 13.2 BETA2: how do debug META_MODE?

[Peter]

On Mon, Feb 20, 2023 at 08:44:59PM -0800, Mark Millard wrote:
! Peter  wrote on
! Date: Tue, 21 Feb 2023 03:45:12 UTC :
! 
! > on /some/ of my nodes, META_MODE seems not being honored anymore:
! > I had to build them another time, and the lengthy lib/clang gets
! > built all over again (tried two times).
! > This is so since 13.2 (BETA2). It did work in 13.1 (RELENG), at least
! > according to the timing from the logfiles. 
! > 
! > Now I'm trying to figure out the difference, because I have some
! > nodes where it appears to more-or-less work (have seen buildworld
! > take 5 minutes), and others where it doesn't (take an hour to build).
! > The thing is scripted, so it is not so very likely an operator error
! > (while not impossible either).
! > 
! > But it seems difficult to figure out details: "make -n" seems to not
! > care about META_MODE, while META_MODE suppresses all useful output from
! > make. And the docs say there are *.meta files (yes there are), but no
! > info about how to verify their content, or how to get make tell what
! > it is going to do and why (and the buildworld is not the most easy
! > to understand target)...
! > 
! > So, some inspiration would be welcome...
! 
! On thing to check on is if filemon.ko is loaded and operational.
! META_MODE greatly depends on it.

That should be the case - 'kldstat' shows it (and I've seen warnings
where it didn't).

! Another thing to know is that the following are very different
! for what all is built  for the "(again #0)" line vs. the other
! two "again" lines, using buildworld as an example context.
! Imagine here the the first buildworld rebuilds llvm/clang
! materials.
! 
! # cd /usr/src/
! # env WITH_META_MODE=yes make buildworld
! # env WITH_META_MODE=yes make installworld
! # env WITH_META_MODE=yes make buildworld (again #0)
! ## no more rebuilds below?
! # env WITH_META_MODE=yes make buildworld (again #1)
! # env WITH_META_MODE=yes make buildworld (again #2)

But what is the difference between #0 and #1?
I am actually grabbing src and obj and zfs-send them away
to the actual install target, then installworld to DESTDIR
which becomes the new root for the next update cycle.
Jails are a bit different because not all of them get the
compiler installed - and jails do have the problem now: if
they get the compiler, it always rebuilt.

According to the ./UPDATING paper, one should do build + install
+ build-again + install-again. That is probably outdated, but on
Release-Changes I'm doing it that way.
During the build-again step I noticed that the META_MODE did already
work, things were fast - but only for the base nodes, not for the
jails.

! Unfortunately, the some of the install activity registers
! as activity that is to cause later rebuild activity:
! updated file dates.

Okay. I tried to find the offensive timestamps, and the problem
is: there aren't any. :(
I grabbed one node that does build fast as expected, and one that
doesn't, and put the single-core output of buildworld side-by-side.
and this is where the difference starts:

[fast builds]
===> kerberos5/tools/make-roken (obj,all,install)
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.c
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.o
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/_proginstall
===> kerberos5/lib/libroken (obj,all,install)
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/libroken.a
building static roken library
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/_libinstall
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/_INCSINS
===> kerberos5/lib/libvers (obj,all,install)
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libvers/libvers.a
building static vers library

[rebuilds it all]
===> kerberos5/tools/make-roken (obj,all,install)
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.c
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken.o
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/make-roken
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/tools/make-roken/_proginstall
===> kerberos5/lib/libroken (obj,all,install)
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/roken.h
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/base64.o
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/copyhostent.o
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/ecalloc.o
Building 
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/lib/libroken/emalloc.o
...

Looking at the first concerned file,
/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/kerberos5/li

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 04:55, Peter  wrote:


> On Mon, Feb 20, 2023 at 08:44:59PM -0800, Mark Millard wrote:
> ! Peter  wrote on
> ! Date: Tue, 21 Feb 2023 03:45:12 UTC :
> ! 
> ! > on /some/ of my nodes, META_MODE seems not being honored anymore:
> ! > I had to build them another time, and the lengthy lib/clang gets
> ! > built all over again (tried two times).
> ! > This is so since 13.2 (BETA2). It did work in 13.1 (RELENG), at least
> ! > according to the timing from the logfiles. 
> ! > 
> ! > Now I'm trying to figure out the difference, because I have some
> ! > nodes where it appears to more-or-less work (have seen buildworld
> ! > take 5 minutes), and others where it doesn't (take an hour to build).
> ! > The thing is scripted, so it is not so very likely an operator error
> ! > (while not impossible either).
> ! > 
> ! > But it seems difficult to figure out details: "make -n" seems to not
> ! > care about META_MODE, while META_MODE suppresses all useful output from
> ! > make. And the docs say there are *.meta files (yes there are), but no
> ! > info about how to verify their content, or how to get make tell what
> ! > it is going to do and why (and the buildworld is not the most easy
> ! > to understand target)...
> ! > 
> ! > So, some inspiration would be welcome...
> ! 
> ! On thing to check on is if filemon.ko is loaded and operational.
> ! META_MODE greatly depends on it.
> 
> That should be the case - 'kldstat' shows it (and I've seen warnings
> where it didn't).
> 
> ! Another thing to know is that the following are very different
> ! for what all is built  for the "(again #0)" line vs. the other
> ! two "again" lines, using buildworld as an example context.
> ! Imagine here the the first buildworld rebuilds llvm/clang
> ! materials.
> ! 
> ! # cd /usr/src/
> ! # env WITH_META_MODE=yes make buildworld
> ! # env WITH_META_MODE=yes make installworld
> ! # env WITH_META_MODE=yes make buildworld (again #0)
> ! ## no more rebuilds below?
> ! # env WITH_META_MODE=yes make buildworld (again #1)
> ! # env WITH_META_MODE=yes make buildworld (again #2)
> 
> But what is the difference between #0 and #1?

awk, cp, ln, rm, sed, and many more from
. . ./tmp/legacy/usr/sbin/have new dates
for rebuilds after installworld (that targets
the running system). Not true for #1 and #2.

The dates on these tools being more recent than
the files that they were involved in producing
leads to rebuilding those files. That in turn
leads to other files being rebuilt.

make with -dM reports the likes of:

   file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...

explicitly as it goes. As I remember tmp/legacy/usr/sbin/
was always part of the path for what I found.

One still has to trace back to were rebuild a rebuild
is not due to something rebuilt in earlier in the same
build. Noting that tmp/legacy/usr/sbin/awk is reported
as newer than its target, leaves the question of how
it ended up being newer: earlier in same build vs.
before build activity? It too must be traced back
to something based on just material from prior to
the build in question.

Note that the above make sequence was only intended
for showing the dependency, not as instructions for a
normal update sequence.

> . . .
> 
> ! See:
> ! 
> ! https://lists.freebsd.org/pipermail/freebsd-current/2021-January/078488.html

This (and later messages in the thread) are about the
"awk, cp, ln, rm, sed, and many more" that make with -dM
explicitly reports (likely from tmp/legacy/usr/sbin/ ).
If you trust the make date comparisons, it is the easiest
way to find out what has "is newer than the target" status
that leads to starting a rebuild sequence. (Other dependent
things then rebuild based on this rebuild. One still has
to trace back to where things start.)

I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
ended up being newer than such a target and, so, causing a
rebuild of that target. I was going the direction: that
it is newer really is unlikely to justify the rebuild for
the target(s) in question. The other direction about how
it got to be newer is also relevant.

> ! 
> ! and:
> ! 
> ! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257616 .
> 
> Thank You, that's exactly the inspiration I was looking for!
> Diving back in...

===
Mark Millard
marklmi at yahoo.com

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 11:56, Mark Millard  wrote:

> On Feb 21, 2023, at 04:55, Peter  wrote:
> 
>> On Mon, Feb 20, 2023 at 08:44:59PM -0800, Mark Millard wrote:
>> ! Peter  wrote on
>> ! Date: Tue, 21 Feb 2023 03:45:12 UTC :
>> ! 
>> ! > on /some/ of my nodes, META_MODE seems not being honored anymore:
>> ! > I had to build them another time, and the lengthy lib/clang gets
>> ! > built all over again (tried two times).
>> ! > This is so since 13.2 (BETA2). It did work in 13.1 (RELENG), at least
>> ! > according to the timing from the logfiles. 
>> ! > 
>> ! > Now I'm trying to figure out the difference, because I have some
>> ! > nodes where it appears to more-or-less work (have seen buildworld
>> ! > take 5 minutes), and others where it doesn't (take an hour to build).
>> ! > The thing is scripted, so it is not so very likely an operator error
>> ! > (while not impossible either).
>> ! > 
>> ! > But it seems difficult to figure out details: "make -n" seems to not
>> ! > care about META_MODE, while META_MODE suppresses all useful output from
>> ! > make. And the docs say there are *.meta files (yes there are), but no
>> ! > info about how to verify their content, or how to get make tell what
>> ! > it is going to do and why (and the buildworld is not the most easy
>> ! > to understand target)...
>> ! > 
>> ! > So, some inspiration would be welcome...
>> ! 
>> ! On thing to check on is if filemon.ko is loaded and operational.
>> ! META_MODE greatly depends on it.
>> 
>> That should be the case - 'kldstat' shows it (and I've seen warnings
>> where it didn't).
>> 
>> ! Another thing to know is that the following are very different
>> ! for what all is built  for the "(again #0)" line vs. the other
>> ! two "again" lines, using buildworld as an example context.
>> ! Imagine here the the first buildworld rebuilds llvm/clang
>> ! materials.
>> ! 
>> ! # cd /usr/src/
>> ! # env WITH_META_MODE=yes make buildworld
>> ! # env WITH_META_MODE=yes make installworld
>> ! # env WITH_META_MODE=yes make buildworld (again #0)
>> ! ## no more rebuilds below?
>> ! # env WITH_META_MODE=yes make buildworld (again #1)
>> ! # env WITH_META_MODE=yes make buildworld (again #2)
>> 
>> But what is the difference between #0 and #1?
> 
> awk, cp, ln, rm, sed, and many more from
> . . ./tmp/legacy/usr/sbin/have new dates
> for rebuilds after installworld (that targets
> the running system). Not true for #1 and #2.
> 
> The dates on these tools being more recent than
> the files that they were involved in producing
> leads to rebuilding those files. That in turn
> leads to other files being rebuilt.
> 
> make with -dM reports the likes of:
> 
>   file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
> 
> explicitly as it goes. As I remember tmp/legacy/usr/sbin/
> was always part of the path for what I found.
> 
> One still has to trace back to were rebuild a rebuild
> is not due to something rebuilt in earlier in the same
> build. Noting that tmp/legacy/usr/sbin/awk is reported
> as newer than its target, leaves the question of how
> it ended up being newer: earlier in same build vs.
> before build activity? It too must be traced back
> to something based on just material from prior to
> the build in question.
> 
> Note that the above make sequence was only intended
> for showing the dependency, not as instructions for a
> normal update sequence.
> 
>> . . .
>> 
>> ! See:
>> ! 
>> ! 
>> https://lists.freebsd.org/pipermail/freebsd-current/2021-January/078488.html
> 
> This (and later messages in the thread) are about the
> "awk, cp, ln, rm, sed, and many more" that make with -dM
> explicitly reports (likely from tmp/legacy/usr/sbin/ ).
> If you trust the make date comparisons, it is the easiest
> way to find out what has "is newer than the target" status
> that leads to starting a rebuild sequence. (Other dependent
> things then rebuild based on this rebuild. One still has
> to trace back to where things start.)
> 
> I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
> ended up being newer than such a target and, so, causing a
> rebuild of that target. I was going the direction: that
> it is newer really is unlikely to justify the rebuild for
> the target(s) in question. The other direction about how
> it got to be newer is also relevant.
> 
>> ! 
>> ! and:
>> ! 
>> ! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257616 .
>> 
>> Thank You, that's exactly the inspiration I was looking for!
>> Diving back in...
> 

I  had forgotten about Simon J. Gerraty's notes in his reply:

https://lists.freebsd.org/pipermail/freebsd-current/2021-January/078628.html

It is about telling META_MODE ignore things that would
otherwise cause rebuild activity. Had I remembered, I would
have also listed it explicitly, not just listing the start
of the thread.

===
Mark Millard
marklmi at yahoo.com

eval: startmsg: not found

[Peter]

https://www.mail-archive.com/stable@freebsd.org/msg00974.html

me too.

13.2-BETA2: xz: illegal instruction

[Peter]

# ./xz < /dev/zero > /dev/null
Illegal instruction (core dumped)

CPU: Intel(R) Core(TM) i3 CPU 540  @ 3.07GHz (3059.08-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x20655  Family=0x6  Model=0x25  Stepping=5

CPUTYPE?=westmere


... investigating...

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 11:56, Mark Millard  wrote:

> On Feb 21, 2023, at 04:55, Peter  wrote:
> 
>> On Mon, Feb 20, 2023 at 08:44:59PM -0800, Mark Millard wrote:
>> ! Peter  wrote on
>> ! Date: Tue, 21 Feb 2023 03:45:12 UTC :
>> ! 
>> ! > on /some/ of my nodes, META_MODE seems not being honored anymore:
>> ! > I had to build them another time, and the lengthy lib/clang gets
>> ! > built all over again (tried two times).
>> ! > This is so since 13.2 (BETA2). It did work in 13.1 (RELENG), at least
>> ! > according to the timing from the logfiles. 
>> ! > 
>> ! > Now I'm trying to figure out the difference, because I have some
>> ! > nodes where it appears to more-or-less work (have seen buildworld
>> ! > take 5 minutes), and others where it doesn't (take an hour to build).
>> ! > The thing is scripted, so it is not so very likely an operator error
>> ! > (while not impossible either).
>> ! > 
>> ! > But it seems difficult to figure out details: "make -n" seems to not
>> ! > care about META_MODE, while META_MODE suppresses all useful output from
>> ! > make. And the docs say there are *.meta files (yes there are), but no
>> ! > info about how to verify their content, or how to get make tell what
>> ! > it is going to do and why (and the buildworld is not the most easy
>> ! > to understand target)...
>> ! > 
>> ! > So, some inspiration would be welcome...
>> ! 
>> ! On thing to check on is if filemon.ko is loaded and operational.
>> ! META_MODE greatly depends on it.
>> 
>> That should be the case - 'kldstat' shows it (and I've seen warnings
>> where it didn't).
>> 
>> ! Another thing to know is that the following are very different
>> ! for what all is built  for the "(again #0)" line vs. the other
>> ! two "again" lines, using buildworld as an example context.
>> ! Imagine here the the first buildworld rebuilds llvm/clang
>> ! materials.
>> ! 
>> ! # cd /usr/src/
>> ! # env WITH_META_MODE=yes make buildworld
>> ! # env WITH_META_MODE=yes make installworld
>> ! # env WITH_META_MODE=yes make buildworld (again #0)
>> ! ## no more rebuilds below?
>> ! # env WITH_META_MODE=yes make buildworld (again #1)
>> ! # env WITH_META_MODE=yes make buildworld (again #2)
>> 
>> But what is the difference between #0 and #1?
> 
> awk, cp, ln, rm, sed, and many more from
> . . ./tmp/legacy/usr/sbin/have new dates
> for rebuilds after installworld (that targets
> the running system). Not true for #1 and #2.
> 
> The dates on these tools being more recent than
> the files that they were involved in producing
> leads to rebuilding those files. That in turn
> leads to other files being rebuilt.
> 
> make with -dM reports the likes of:
> 
>   file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
> 
> explicitly as it goes. As I remember tmp/legacy/usr/sbin/
> was always part of the path for what I found.
> 
> One still has to trace back to were rebuild a rebuild
> is not due to something rebuilt in earlier in the same
> build. Noting that tmp/legacy/usr/sbin/awk is reported
> as newer than its target, leaves the question of how
> it ended up being newer: earlier in same build vs.
> before build activity? It too must be traced back
> to something based on just material from prior to
> the build in question.
> 
> Note that the above make sequence was only intended
> for showing the dependency, not as instructions for a
> normal update sequence.
> 
>> . . .
>> 
>> ! See:
>> ! 
>> ! 
>> https://lists.freebsd.org/pipermail/freebsd-current/2021-January/078488.html
> 
> This (and later messages in the thread) are about the
> "awk, cp, ln, rm, sed, and many more" that make with -dM
> explicitly reports (likely from tmp/legacy/usr/sbin/ ).
> If you trust the make date comparisons, it is the easiest
> way to find out what has "is newer than the target" status
> that leads to starting a rebuild sequence. (Other dependent
> things then rebuild based on this rebuild. One still has
> to trace back to where things start.)
> 
> I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
> ended up being newer than such a target and, so, causing a
> rebuild of that target. I was going the direction: that
> it is newer really is unlikely to justify the rebuild for
> the target(s) in question. The other direction about how
> it got to be newer is also relevant.

Using awk as an example, for the (re)build of awk in:

/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/

in my context, here is a list of files that installworld installs
that contribute to the build (and a couple of "Fork" lines):

(from 
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/usr.bin/awk/awk.full.meta
 )

E 51580 /bin/sh
R 51580 /etc/libmap.conf
R 51580 /usr/local/etc/libmap.d
R 51580 /usr/local/etc/libmap.d/mesa.conf
R 51580 /var/run/ld-elf.so.hints
R 51580 /lib/libedit.so.8
R 51580 /lib/libc.so.7
R 51580 /lib/libtinfow.so.9
R 51580 /usr/share/locale/C.UTF-8/LC_CTYPE
F 51580 51585
E 51585 /usr/bin/cc
R 51585 /etc/libmap.conf

Re: 13.2-BETA2: xz: illegal instruction

[Peter]

> # ./xz < /dev/zero > /dev/null
> Illegal instruction (core dumped)

> CPU: Intel(R) Core(TM) i3 CPU 540  @ 3.07GHz (3059.08-MHz K8-class 
> CPU)
> Origin="GenuineIntel"  Id=0x20655  Family=0x6  Model=0x25  Stepping=5 

> CPUTYPE?=westmere

It's liblzma, and it doesn't want westmere, it apparently wants
penryn.

Re: 13.2-BETA2: xz: illegal instruction

[Peter]

>> CPU: Intel(R) Core(TM) i3 CPU 540  @ 3.07GHz (3059.08-MHz K8-class 
>> CPU)
>> Origin="GenuineIntel"  Id=0x20655  Family=0x6  Model=0x25  Stepping=5
>>  

>> CPUTYPE?=westmere

> It's liblzma, and it doesn't want westmere, it apparently wants
> penryn.

Hm *grumble*.

According to https://en.wikipedia.org/wiki/Clarkdale_(microprocessor)
the i3-5xx is Clarkdale, based on Westmere. And according to
https://en.wikipedia.org/wiki/Westmere_(microarchitecture)
Clarkdale is Westmere.

Trying Nehalem - seems to work. Nehalem is 45nm, this chip is 32nm.
Well then, rebuilding for Nehalem...

Re: stable/13 snapshot's /etc/rc.d/machine_id has use of main's startmsg from /etc/rc.subr so it reports 2 "eval: startmsg: not found"

[Zhenlei Huang]

Hi,

`libexec/rc/rc.d/machine_id` (https://reviews.freebsd.org/D37722 
) is MFC'ed into stable/13 but its 
precedence https://reviews.freebsd.org/D34514 
 is not.

So we should either MFC D34514, or change libexec/rc/rc.d/machine_id in 
stable/13 to use `check_startmsgs`.

I'd personally prefer MFCing ;)

Best regards,
Zhenlei

> On Jan 22, 2023, at 11:32 AM, Mark Millard  wrote:
> 
> When I booted a new stable/13 snapshot install, the messaging
> included:
> 
> . . .
> Updating motd:.
> eval: startmsg: not found
> eval: startmsg: not found
> Clearing /tmp (X related).
> . . .
> 
> It looks like the "eval: startmsg: not found" lines
> are from:
> 
> # grep -r "\" /etc/ 
> /etc/rc.d/machine_id:   startmsg -n "Creating ${machine_id_file} "
> /etc/rc.d/machine_id:   startmsg 'done.'
> 
> (No more matches found.)
> 
> The following was not found on stable/13:
> 
> /etc/rc.subr:# startmsg
> /etc/rc.subr:startmsg()
> /etc/rc.subr:   startmsg "Starting ${name}."
> 
> 
> ===
> Mark Millard
> marklmi at yahoo.com
> 
> 

Re: stable/13 snapshot's /etc/rc.d/machine_id has use of main's startmsg from /etc/rc.subr so it reports 2 "eval: startmsg: not found"

[Zhenlei Huang]

Sorry for missing the author tcberner@ of `libexec/rc/rc.d/machine_id`.

> On Feb 22, 2023, at 9:25 AM, Zhenlei Huang  wrote:
> 
> Hi,
> 
> `libexec/rc/rc.d/machine_id` (https://reviews.freebsd.org/D37722 
> ) is MFC'ed into stable/13 but its 
> precedence https://reviews.freebsd.org/D34514 
>  is not.
> 
> So we should either MFC D34514, or change libexec/rc/rc.d/machine_id in 
> stable/13 to use `check_startmsgs`.
> 
> I'd personally prefer MFCing ;)
> 
> Best regards,
> Zhenlei
> 
>> On Jan 22, 2023, at 11:32 AM, Mark Millard > > wrote:
>> 
>> When I booted a new stable/13 snapshot install, the messaging
>> included:
>> 
>> . . .
>> Updating motd:.
>> eval: startmsg: not found
>> eval: startmsg: not found
>> Clearing /tmp (X related).
>> . . .
>> 
>> It looks like the "eval: startmsg: not found" lines
>> are from:
>> 
>> # grep -r "\" /etc/ 
>> /etc/rc.d/machine_id:   startmsg -n "Creating ${machine_id_file} "
>> /etc/rc.d/machine_id:   startmsg 'done.'
>> 
>> (No more matches found.)
>> 
>> The following was not found on stable/13:
>> 
>> /etc/rc.subr:# startmsg
>> /etc/rc.subr:startmsg()
>> /etc/rc.subr:   startmsg "Starting ${name}."
>> 
>> 
>> ===
>> Mark Millard
>> marklmi at yahoo.com 
>> 
>> 
> 
> 
> 

Re: 13.2 BETA2: how do debug META_MODE?

[Peter]

On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
! On Feb 21, 2023, at 04:55, Peter  wrote:
! 
! > ! # cd /usr/src/
! > ! # env WITH_META_MODE=yes make buildworld
! > ! # env WITH_META_MODE=yes make installworld
! > ! # env WITH_META_MODE=yes make buildworld (again #0)
! > ! ## no more rebuilds below?
! > ! # env WITH_META_MODE=yes make buildworld (again #1)
! > ! # env WITH_META_MODE=yes make buildworld (again #2)
! > 
! > But what is the difference between #0 and #1?
! 
! awk, cp, ln, rm, sed, and many more from
! . . ./tmp/legacy/usr/sbin/have new dates
! for rebuilds after installworld (that targets
! the running system). Not true for #1 and #2.
! 
! The dates on these tools being more recent than
! the files that they were involved in producing
! leads to rebuilding those files. That in turn
! leads to other files being rebuilt.
! 
! make with -dM reports the likes of:
! 
!file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
! 
! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
! was always part of the path for what I found.

Mark, thanks a lot for the proper input at the right time!

This put me on the right track and I mananged to analyze and
understand what is actually happening.

It looks like my issue does resolve itself somehow, and things
start  to behave as expected again after four builds.

! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
! ended up being newer than such a target and, so, causing a
! rebuild of that target. I was going the direction: that
! it is newer really is unlikely to justify the rebuild for
! the target(s) in question. The other direction about how
! it got to be newer is also relevant.

I have now analyzed some parts of it. META_MODE typically finds some
build-tools to rebuild, but then if the result is not different
from what was there before, then "install" will not copy it to the
bin-dir, and so the avalanche gets usually avoided.

[analysis] Re: 13.2 BETA2: how do debug META_MODE?

[Peter]

It appears as my issue goes away by itself at the fourth subsequent
build.

># cd /usr/src/
># env WITH_META_MODE=yes make buildworld
># env WITH_META_MODE=yes make installworld
># env WITH_META_MODE=yes make buildworld (again #0)
>## no more rebuilds below?
># env WITH_META_MODE=yes make buildworld (again #1)
># env WITH_META_MODE=yes make buildworld (again #2)
>
>"again #0" will rebuild llvm/clang. The other two "again"s
>will not.

Strangely, I observed rather the opposite: the issue was with those
nodes that do *not* get installed. Those that do get installed, they
behaved as expected.

Details:

When starting a build, there are some programs in
/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/ from the last build.
These might be suitable for the new build (or they might not, who
knows) - but then, very early in the buildworld, many of these files
are copied from the running base system, with their mtime preserved.
So now these files do not have an mtime of the last build, but of the
last *install* of this running instance. (And that may be too late.)
This is done here - and I have no idea how these files are selected:

> --
> >>> Rebuilding the temporary build tree
> --
[...]
> Building /usr/obj/usr/src/amd64.amd64/tools/build/host-symlinks
> Linking host tools into /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin

Over all, /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/ tends to become
a zoo: some files are copied from the base system, some are installed
during the build, some are not replaced at all.

I tried to figure what exactly is happening, with the example of
the make-roken + roken.h.
I start the upgrade to 13.2 with an empty obj tree (because after
switching and rebasing branches, my mtimes are recreated from the
latest commit - and they will certainly not align with the old ones).

In the first build the binaries in the obj tree have to be created.
But since the running system is still 13.1, these binaries get all
stamped for 13.1.

In the second build -now running 13.2- META_MODE decides that the
mtimes are now fine and does not rebuild.
META_MODE has no notion of the currently running OS version, it only
considers mtimes and cannot detect that these dependencies are stamped
with an old OS version.

In the third build META_MODE finds some mtimes obsolete again, and
does rebuild toolchain binaries. This would not have any effect
because "install" would not copy them to the bin-dir if the same
binaries are already there. But in this scenario the old binaries
are from 13.1, so they are different, and "install" copies the new
ones in - and now everything that somehow depends on them will also
rebuild.

Finally in the fourth build everything appears to be fine.

Conclusion:
For a safe upgrade (specifically for a major version change) it is
not so much necessary to delete the obj tree before the first build,
but rather *after* the first build, i.e. after the base system has
been upgraded.

Some timings:

The base gets installed into a clean DESTDIR and used for the next
pass. The obj-trees are individually kept.

Initial (obj trees deleted)
( 4 vcore)
230217231308.base.pass1.sst: real 9339.64   base w/ kernels
230217231308.base.pass2.sst: real 7982.69
230217231308.admn.pass1.jail.sst:real 9346.90   jail w/ compiler
230217231308.admn.pass2.jail.sst:real 5460.16
230217231308.data.pass1.jail.sst:real 4094.04   jail w/o compiler
230217231308.data.pass2.jail.sst:real 143.39
230217231308.iamk.pass1.jail.sst:real 8050.27   jail w/ compiler
230217231308.iamk.pass2.jail.sst:real 5226.32
230217231308.oper.pass1.jail.sst:real 2910.28   jail w/o compiler
230217231308.oper.pass2.jail.sst:real 92.05
230217231308.rail.pass1.jail.sst:real 3236.29   jail w/o compiler
230217231308.rail.pass2.jail.sst:real 99.49
230217231308.tele.pass1.jail.sst:real 3170.34   jail w/o compiler
230217231308.tele.pass2.jail.sst:real 180.65

pass3
(10 vcore)
230222000242.base.std.sst: real 1162.80 base w/ kernels
230222000242.admn.std.jail.sst:real 1759.15 jail w/ compiler
230222000242.data.std.jail.sst:real 155.54  jail w/o compiler
230222000242.iamk.std.jail.sst:real 1715.07 jail w/ compiler
230222000242.oper.std.jail.sst:real 149.51  jail w/o compiler
230222000242.rail.std.jail.sst:real 151.73  jail w/o compiler
230222000242.tele.std.jail.sst:real 150.52  jail w/o compiler

pass4
(10 vcore)
230222021535.edge.std.sst: real 1018.79 base w/ kernels
230222021535.admn.std.jail.sst:real 101.61  jail w/ compiler
230222021535.data.std.jail.sst:real 67.47   jail w/o compiler
230222021535.iamk.std.jail.sst:real 100.91  jail w/ compiler
230222021535.oper.std.jail.sst:real 66.52   jail w/o compiler
230222021535.rail.std.jail.sst:real 68.00   jail w/o compiler
230222021535.tele.std.jail.sst:real 66.54   jail w/o compiler

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 18:10, Peter  wrote:

> On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
> ! On Feb 21, 2023, at 04:55, Peter  wrote:
> ! 
> ! > ! # cd /usr/src/
> ! > ! # env WITH_META_MODE=yes make buildworld
> ! > ! # env WITH_META_MODE=yes make installworld
> ! > ! # env WITH_META_MODE=yes make buildworld (again #0)
> ! > ! ## no more rebuilds below?
> ! > ! # env WITH_META_MODE=yes make buildworld (again #1)
> ! > ! # env WITH_META_MODE=yes make buildworld (again #2)
> ! > 
> ! > But what is the difference between #0 and #1?
> ! 
> ! awk, cp, ln, rm, sed, and many more from
> ! . . ./tmp/legacy/usr/sbin/have new dates
> ! for rebuilds after installworld (that targets
> ! the running system). Not true for #1 and #2.
> ! 
> ! The dates on these tools being more recent than
> ! the files that they were involved in producing
> ! leads to rebuilding those files. That in turn
> ! leads to other files being rebuilt.
> ! 
> ! make with -dM reports the likes of:
> ! 
> !file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
> ! 
> ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
> ! was always part of the path for what I found.
> 
> Mark, thanks a lot for the proper input at the right time!
> 
> This put me on the right track and I mananged to analyze and
> understand what is actually happening.
> 
> It looks like my issue does resolve itself somehow, and things
> start  to behave as expected again after four builds.

Intersting.

> ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
> ! ended up being newer than such a target and, so, causing a
> ! rebuild of that target. I was going the direction: that
> ! it is newer really is unlikely to justify the rebuild for
> ! the target(s) in question. The other direction about how
> ! it got to be newer is also relevant.
> 
> I have now analyzed some parts of it. META_MODE typically finds some
> build-tools to rebuild, but then if the result is not different
> from what was there before, then "install" will not copy it to the
> bin-dir, and so the avalanche gets usually avoided.
> 

The implication is that "install -C" is in use, quoting the
man page:

 -C  Copy the file.  If the target file already exists and the files
 are the same, then do not change the modification time of the
 target.  If the target's file flags and mode need not to be
 changed, the target's inode change time is also unchanged.

 -c  Copy the file.  This is actually the default.  The -c option is
 only included for backwards compatibility.

-C might have more of an effect in a reproducible-build
style build process than on a non-reproducible-build
style one.


===
Mark Millard
marklmi at yahoo.com

Re: 13.2 BETA2: how do debug META_MODE?

[Peter]

On Tue, Feb 21, 2023 at 06:44:09PM -0800, Mark Millard wrote:
! On Feb 21, 2023, at 18:10, Peter  wrote:
! 
! > On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
! > ! On Feb 21, 2023, at 04:55, Peter  wrote:
! > ! 
! > ! > ! # cd /usr/src/
! > ! > ! # env WITH_META_MODE=yes make buildworld
! > ! > ! # env WITH_META_MODE=yes make installworld
! > ! > ! # env WITH_META_MODE=yes make buildworld (again #0)
! > ! > ! ## no more rebuilds below?
! > ! > ! # env WITH_META_MODE=yes make buildworld (again #1)
! > ! > ! # env WITH_META_MODE=yes make buildworld (again #2)
! > ! > 
! > ! > But what is the difference between #0 and #1?
! > ! 
! > ! awk, cp, ln, rm, sed, and many more from
! > ! . . ./tmp/legacy/usr/sbin/have new dates
! > ! for rebuilds after installworld (that targets
! > ! the running system). Not true for #1 and #2.
! > ! 
! > ! The dates on these tools being more recent than
! > ! the files that they were involved in producing
! > ! leads to rebuilding those files. That in turn
! > ! leads to other files being rebuilt.
! > ! 
! > ! make with -dM reports the likes of:
! > ! 
! > !file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
! > ! 
! > ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
! > ! was always part of the path for what I found.
! > 
! > Mark, thanks a lot for the proper input at the right time!
! > 
! > This put me on the right track and I mananged to analyze and
! > understand what is actually happening.
! > 
! > It looks like my issue does resolve itself somehow, and things
! > start  to behave as expected again after four builds.
! 
! Intersting.
! 
! > ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
! > ! ended up being newer than such a target and, so, causing a
! > ! rebuild of that target. I was going the direction: that
! > ! it is newer really is unlikely to justify the rebuild for
! > ! the target(s) in question. The other direction about how
! > ! it got to be newer is also relevant.
! > 
! > I have now analyzed some parts of it. META_MODE typically finds some
! > build-tools to rebuild, but then if the result is not different
! > from what was there before, then "install" will not copy it to the
! > bin-dir, and so the avalanche gets usually avoided.
! > 
! 
! The implication is that "install -C" is in use, quoting the
! man page:
! 
!  -C  Copy the file.  If the target file already exists and the files
!  are the same, then do not change the modification time of the
!  target.  If the target's file flags and mode need not to be
!  changed, the target's inode change time is also unchanged.
! 
!  -c  Copy the file.  This is actually the default.  The -c option is
!  only included for backwards compatibility.
! 
! -C might have more of an effect in a reproducible-build
! style build process than on a non-reproducible-build
! style one.

Yepp. "install -p" is used, see /usr/src/tools/install.sh

Re: [analysis] 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 18:32, Peter  wrote:

> It appears as my issue goes away by itself at the fourth subsequent
> build.
> 
>> # cd /usr/src/
>> # env WITH_META_MODE=yes make buildworld
>> # env WITH_META_MODE=yes make installworld
>> # env WITH_META_MODE=yes make buildworld (again #0)
>> ## no more rebuilds below?
>> # env WITH_META_MODE=yes make buildworld (again #1)
>> # env WITH_META_MODE=yes make buildworld (again #2)
>> 
>> "again #0" will rebuild llvm/clang. The other two "again"s
>> will not.
> 
> Strangely, I observed rather the opposite: the issue was with those
> nodes that do *not* get installed. Those that do get installed, they
> behaved as expected.

My older notes from 2021 were better at being explicit
about the initial context assumed:

QUOTE
Given an already built, installed and booted system version, I've
noted a big difference for META_MODE in 2 rebuild contexts (no
source updates involved)
END QUOTE

In other words, all the build activity was rebuild-same-version
activity or reinstall-same-version activity in my example
sequence.

So I expect that the attempted comparison happens to not have a
matching context to make the comparison with.

> Details:
> 
> When starting a build, there are some programs in
> /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/ from the last build.
> These might be suitable for the new build (or they might not, who
> knows) - but then, very early in the buildworld, many of these files
> are copied from the running base system, with their mtime preserved.
> So now these files do not have an mtime of the last build, but of the
> last *install* of this running instance. (And that may be too late.)
> This is done here - and I have no idea how these files are selected:
> 
>> --
> Rebuilding the temporary build tree
>> --
> [...]
>> Building /usr/obj/usr/src/amd64.amd64/tools/build/host-symlinks
>> Linking host tools into /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin
> 
> Over all, /usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/ tends to become
> a zoo: some files are copied from the base system, some are installed
> during the build, some are not replaced at all.
> 
> I tried to figure what exactly is happening, with the example of
> the make-roken + roken.h.
> I start the upgrade to 13.2 with an empty obj tree (because after
> switching and rebasing branches, my mtimes are recreated from the
> latest commit - and they will certainly not align with the old ones).
> 
> In the first build the binaries in the obj tree have to be created.
> But since the running system is still 13.1, these binaries get all
> stamped for 13.1.
> 
> In the second build -now running 13.2- META_MODE decides that the
> mtimes are now fine and does not rebuild.
> META_MODE has no notion of the currently running OS version, it only
> considers mtimes and cannot detect that these dependencies are stamped
> with an old OS version.

In terms of your example, in case the earlier was unclear:

In my test sequence, 13.2 would have already been built, installed,
and booted before the buildworld/installworld sequence that I
listed. So my notes do not match your full sequence. They are more
of a rebuild-already-existing sequence to analyze the behavior
just for that simpler context than a realistic overall normal-build
sequence. Having a installworld is only there to see what
differences it makes compared to later not having such between
such rebuild-already-existing runs. The isntallworld was an
example of reinstall-same-version-as-already-running.

> In the third build META_MODE finds some mtimes obsolete again, and
> does rebuild toolchain binaries. This would not have any effect
> because "install" would not copy them to the bin-dir if the same
> binaries are already there. But in this scenario the old binaries
> are from 13.1, so they are different, and "install" copies the new
> ones in - and now everything that somehow depends on them will also
> rebuild.
> 
> Finally in the fourth build everything appears to be fine.
> 
> Conclusion:
> For a safe upgrade (specifically for a major version change) it is
> not so much necessary to delete the obj tree before the first build,
> but rather *after* the first build, i.e. after the base system has
> been upgraded.
> 
> Some timings:
> 
> The base gets installed into a clean DESTDIR and used for the next
> pass. The obj-trees are individually kept.
> 
> Initial (obj trees deleted)
> ( 4 vcore)
> 230217231308.base.pass1.sst: real 9339.64 base w/ kernels
> 230217231308.base.pass2.sst: real 7982.69
> 230217231308.admn.pass1.jail.sst:real 9346.90 jail w/ compiler
> 230217231308.admn.pass2.jail.sst:real 5460.16
> 230217231308.data.pass1.jail.sst:real 4094.04 jail w/o compiler
> 230217231308.data.pass2.jail.sst:real 143.39
> 230217231308.iamk.pass1.jail.sst:real 8050.27 jail w/ compiler
> 230217231308.iamk.pass2.jail.sst:real 5226.32
> 

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 19:11, Peter  wrote:

> On Tue, Feb 21, 2023 at 06:44:09PM -0800, Mark Millard wrote:
> ! On Feb 21, 2023, at 18:10, Peter  wrote:
> ! 
> ! > On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
> ! > ! On Feb 21, 2023, at 04:55, Peter  wrote:
> ! > ! 
> ! > ! > ! # cd /usr/src/
> ! > ! > ! # env WITH_META_MODE=yes make buildworld
> ! > ! > ! # env WITH_META_MODE=yes make installworld
> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #0)
> ! > ! > ! ## no more rebuilds below?
> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #1)
> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #2)
> ! > ! > 
> ! > ! > But what is the difference between #0 and #1?
> ! > ! 
> ! > ! awk, cp, ln, rm, sed, and many more from
> ! > ! . . ./tmp/legacy/usr/sbin/have new dates
> ! > ! for rebuilds after installworld (that targets
> ! > ! the running system). Not true for #1 and #2.
> ! > ! 
> ! > ! The dates on these tools being more recent than
> ! > ! the files that they were involved in producing
> ! > ! leads to rebuilding those files. That in turn
> ! > ! leads to other files being rebuilt.
> ! > ! 
> ! > ! make with -dM reports the likes of:
> ! > ! 
> ! > !file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
> ! > ! 
> ! > ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
> ! > ! was always part of the path for what I found.
> ! > 
> ! > Mark, thanks a lot for the proper input at the right time!
> ! > 
> ! > This put me on the right track and I mananged to analyze and
> ! > understand what is actually happening.
> ! > 
> ! > It looks like my issue does resolve itself somehow, and things
> ! > start  to behave as expected again after four builds.
> ! 
> ! Intersting.
> ! 
> ! > ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
> ! > ! ended up being newer than such a target and, so, causing a
> ! > ! rebuild of that target. I was going the direction: that
> ! > ! it is newer really is unlikely to justify the rebuild for
> ! > ! the target(s) in question. The other direction about how
> ! > ! it got to be newer is also relevant.
> ! > 
> ! > I have now analyzed some parts of it. META_MODE typically finds some
> ! > build-tools to rebuild, but then if the result is not different
> ! > from what was there before, then "install" will not copy it to the
> ! > bin-dir, and so the avalanche gets usually avoided.
> ! > 
> ! 
> ! The implication is that "install -C" is in use, quoting the
> ! man page:
> ! 
> !  -C  Copy the file.  If the target file already exists and the files
> !  are the same, then do not change the modification time of the
> !  target.  If the target's file flags and mode need not to be
> !  changed, the target's inode change time is also unchanged.
> ! 
> !  -c  Copy the file.  This is actually the default.  The -c option is
> !  only included for backwards compatibility.
> ! 
> ! -C might have more of an effect in a reproducible-build
> ! style build process than on a non-reproducible-build
> ! style one.
> 
> Yepp. "install -p" is used, see /usr/src/tools/install.sh
> 

The code for the _bootstap_tools_links uses "cp -pf",
not install, to establish part of . . ./tmp/legacy/bin/ .
(Note: . . ./tmp/legacy/sbin -> ../bin so is a via a
symbolic link.) Before the "cp -pf" there is a "rm -f"
deleting the target file before the copy: the prior
file in . . ./tmp/legacy/bin/ is never directly
preserved. (The new copy might still be identical
to the old one: the source path one might happen to
be identical as well.)

# Link the tools that we need for building but don't need to bootstrap because
# the host version is known to be compatible into ${WORLDTMP}/legacy
# We do this before building any of the bootstrap tools in case they depend on
# the presence of any of the links (e.g. as m4/lex/awk)
${_bt}-links: .PHONY

.for _tool in ${_bootstrap_tools_links}
${_bt}-link-${_tool}: .PHONY
@rm -f "${WORLDTMP}/legacy/bin/${_tool}"; \
source_path=`which ${_tool}`; \
if [ ! -e "$${source_path}" ] ; then \
echo "Cannot find host tool '${_tool}'"; false; \
fi; \
cp -pf "$${source_path}" "${WORLDTMP}/legacy/bin/${_tool}"
${_bt}-links: ${_bt}-link-${_tool}
.endfor

Note: This is for the !defined(BOOTSTRAP_ALL_TOOLS) case.
Note: the code uses the abbreviation: _bt=_bootstrap-tools

_bootstrap_tools_links is built mostly in terms of
_basic_bootstrap_tools and _basic_bootstrap_tools_multilink from
earlier logic. For reference, showing what ends up handled
this way:

# grep -r "_bootstrap_tools" /usr/main-src/Makefile* /usr/main-src/share/ | more
/usr/main-src/Makefile.inc1:# _bootstrap_tools_links variable.
/usr/main-src/Makefile.inc1:_bootstrap_tools_links+=m4 lex
/usr/main-src/Makefile.inc1:_bootstrap_tools_links+=mtree
/usr/main-src/Makefile.inc1:_bootstrap_tools_links+=cat
/usr/main-src/Makefile.inc1:_

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 20:51, Mark Millard  wrote:

> On Feb 21, 2023, at 19:11, Peter  wrote:
> 
>> On Tue, Feb 21, 2023 at 06:44:09PM -0800, Mark Millard wrote:
>> ! On Feb 21, 2023, at 18:10, Peter  wrote:
>> ! 
>> ! > On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
>> ! > ! On Feb 21, 2023, at 04:55, Peter  wrote:
>> ! > ! 
>> ! > ! > ! # cd /usr/src/
>> ! > ! > ! # env WITH_META_MODE=yes make buildworld
>> ! > ! > ! # env WITH_META_MODE=yes make installworld
>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #0)
>> ! > ! > ! ## no more rebuilds below?
>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #1)
>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #2)
>> ! > ! > 
>> ! > ! > But what is the difference between #0 and #1?
>> ! > ! 
>> ! > ! awk, cp, ln, rm, sed, and many more from
>> ! > ! . . ./tmp/legacy/usr/sbin/have new dates
>> ! > ! for rebuilds after installworld (that targets
>> ! > ! the running system). Not true for #1 and #2.
>> ! > ! 
>> ! > ! The dates on these tools being more recent than
>> ! > ! the files that they were involved in producing
>> ! > ! leads to rebuilding those files. That in turn
>> ! > ! leads to other files being rebuilt.
>> ! > ! 
>> ! > ! make with -dM reports the likes of:
>> ! > ! 
>> ! > !file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
>> ! > ! 
>> ! > ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
>> ! > ! was always part of the path for what I found.
>> ! > 
>> ! > Mark, thanks a lot for the proper input at the right time!
>> ! > 
>> ! > This put me on the right track and I mananged to analyze and
>> ! > understand what is actually happening.
>> ! > 
>> ! > It looks like my issue does resolve itself somehow, and things
>> ! > start  to behave as expected again after four builds.
>> ! 
>> ! Intersting.
>> ! 
>> ! > ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
>> ! > ! ended up being newer than such a target and, so, causing a
>> ! > ! rebuild of that target. I was going the direction: that
>> ! > ! it is newer really is unlikely to justify the rebuild for
>> ! > ! the target(s) in question. The other direction about how
>> ! > ! it got to be newer is also relevant.
>> ! > 
>> ! > I have now analyzed some parts of it. META_MODE typically finds some
>> ! > build-tools to rebuild, but then if the result is not different
>> ! > from what was there before, then "install" will not copy it to the
>> ! > bin-dir, and so the avalanche gets usually avoided.
>> ! > 
>> ! 
>> ! The implication is that "install -C" is in use, quoting the
>> ! man page:
>> ! 
>> !  -C  Copy the file.  If the target file already exists and the 
>> files
>> !  are the same, then do not change the modification time of the
>> !  target.  If the target's file flags and mode need not to be
>> !  changed, the target's inode change time is also unchanged.
>> ! 
>> !  -c  Copy the file.  This is actually the default.  The -c option 
>> is
>> !  only included for backwards compatibility.
>> ! 
>> ! -C might have more of an effect in a reproducible-build
>> ! style build process than on a non-reproducible-build
>> ! style one.
>> 
>> Yepp. "install -p" is used, see /usr/src/tools/install.sh

That may be incorrect about what is happening for
_bootstap_tools_links and other things. Why do I
say that? Several points . . .

I do not see "tools" in any PATH= so far, making implicit
use unlikely.

/usr/main-src/share/mk/sys.mk:INSTALL   ?=  ${INSTALL_CMD:Uinstall}
/usr/main-src/share/mk/src.tools.mk:INSTALL_CMD?=   install

vs.

/usr/main-src/Makefile: INSTALL="sh ${.CURDIR}/tools/install.sh"
/usr/main-src/Makefile.inc1:BMAKEENV=   INSTALL="sh 
${.CURDIR}/tools/install.sh" \
/usr/main-src/Makefile.inc1:KTMAKEENV=  INSTALL="sh 
${.CURDIR}/tools/install.sh" \

Also:

# kernel-tools stage
KTMAKEENV=  INSTALL="sh ${.CURDIR}/tools/install.sh" \

vs.

# world stage
WMAKEENV=   ${CROSSENV} \
INSTALL="${INSTALL_CMD} -U" \
and:
.if defined(DB_FROM_SRC) || defined(NO_ROOT)
IMAKE_INSTALL=  INSTALL="${INSTALL_CMD} ${INSTALLFLAGS}"

So: explicitly varying styles for various contexts.

It looks to me like the make output would show:

install . . . 
vs.
sh . . ./tools/install.sh . . .

based on if the script is not-in-use vs. is-in-use
(respectively).

Looking at some of my logs I see  the likes of:

install   -o root -g wheel -m 555   cap_mkdb /FBSDFSSDroot-mnt//usr/bin/cap_mkdb

which looks to not be using the script and not using
either -C or -p as well. cap_mkdb is an example from
the _bootstap_tools_links list.

Some of the below do have -C use, others do not.

# grep -r '\-o.*-g.*-m' /usr/main-src/Makefile* /usr/main-src/share/ | more
/usr/main-src/share/sendmail/Makefile:  ${INSTALL} -T 
package=${PACKAGE:Usendmail} ${TAGS_ARGS} -o ${BINOWN} -g ${BINGRP} -m 755 -d 
${DDIR}/${dir}
/usr/main-src/share/sendmail

Re: 13.2 BETA2: how do debug META_MODE?

[Mark Millard]

On Feb 21, 2023, at 21:53, Mark Millard  wrote:

> On Feb 21, 2023, at 20:51, Mark Millard  wrote:
> 
>> On Feb 21, 2023, at 19:11, Peter  wrote:
>> 
>>> On Tue, Feb 21, 2023 at 06:44:09PM -0800, Mark Millard wrote:
>>> ! On Feb 21, 2023, at 18:10, Peter  wrote:
>>> ! 
>>> ! > On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
>>> ! > ! On Feb 21, 2023, at 04:55, Peter  wrote:
>>> ! > ! 
>>> ! > ! > ! # cd /usr/src/
>>> ! > ! > ! # env WITH_META_MODE=yes make buildworld
>>> ! > ! > ! # env WITH_META_MODE=yes make installworld
>>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #0)
>>> ! > ! > ! ## no more rebuilds below?
>>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #1)
>>> ! > ! > ! # env WITH_META_MODE=yes make buildworld (again #2)
>>> ! > ! > 
>>> ! > ! > But what is the difference between #0 and #1?
>>> ! > ! 
>>> ! > ! awk, cp, ln, rm, sed, and many more from
>>> ! > ! . . ./tmp/legacy/usr/sbin/have new dates
>>> ! > ! for rebuilds after installworld (that targets
>>> ! > ! the running system). Not true for #1 and #2.
>>> ! > ! 
>>> ! > ! The dates on these tools being more recent than
>>> ! > ! the files that they were involved in producing
>>> ! > ! leads to rebuilding those files. That in turn
>>> ! > ! leads to other files being rebuilt.
>>> ! > ! 
>>> ! > ! make with -dM reports the likes of:
>>> ! > ! 
>>> ! > !file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target...
>>> ! > ! 
>>> ! > ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
>>> ! > ! was always part of the path for what I found.
>>> ! > 
>>> ! > Mark, thanks a lot for the proper input at the right time!
>>> ! > 
>>> ! > This put me on the right track and I mananged to analyze and
>>> ! > understand what is actually happening.
>>> ! > 
>>> ! > It looks like my issue does resolve itself somehow, and things
>>> ! > start  to behave as expected again after four builds.
>>> ! 
>>> ! Intersting.
>>> ! 
>>> ! > ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
>>> ! > ! ended up being newer than such a target and, so, causing a
>>> ! > ! rebuild of that target. I was going the direction: that
>>> ! > ! it is newer really is unlikely to justify the rebuild for
>>> ! > ! the target(s) in question. The other direction about how
>>> ! > ! it got to be newer is also relevant.
>>> ! > 
>>> ! > I have now analyzed some parts of it. META_MODE typically finds some
>>> ! > build-tools to rebuild, but then if the result is not different
>>> ! > from what was there before, then "install" will not copy it to the
>>> ! > bin-dir, and so the avalanche gets usually avoided.
>>> ! > 
>>> ! 
>>> ! The implication is that "install -C" is in use, quoting the
>>> ! man page:
>>> ! 
>>> !  -C  Copy the file.  If the target file already exists and the 
>>> files
>>> !  are the same, then do not change the modification time of the
>>> !  target.  If the target's file flags and mode need not to be
>>> !  changed, the target's inode change time is also unchanged.
>>> ! 
>>> !  -c  Copy the file.  This is actually the default.  The -c option 
>>> is
>>> !  only included for backwards compatibility.
>>> ! 
>>> ! -C might have more of an effect in a reproducible-build
>>> ! style build process than on a non-reproducible-build
>>> ! style one.
>>> 
>>> Yepp. "install -p" is used, see /usr/src/tools/install.sh
> 
> That may be incorrect about what is happening for
> _bootstap_tools_links and other things. Why do I
> say that? Several points . . .

I missed looking for a obvious type of evidence:

-rw-r--r--  1 root  wheel  2355 Apr 28 15:20:53 2021 
/usr/main-src/tools/install.sh

The script is not executable, which explains the
use of an explicit sh in: INSTALL="sh ${.CURDIR}/tools/install.sh"

This tends to nail down that the likes of:

install   -o root -g wheel -m 555 . . .

in the output is not an example of using the script.

> I do not see "tools" in any PATH= so far, making implicit
> use unlikely.
> 
> /usr/main-src/share/mk/sys.mk:INSTALL   ?=  
> ${INSTALL_CMD:Uinstall}
> /usr/main-src/share/mk/src.tools.mk:INSTALL_CMD?=   install
> 
> vs.
> 
> /usr/main-src/Makefile: INSTALL="sh ${.CURDIR}/tools/install.sh"
> /usr/main-src/Makefile.inc1:BMAKEENV=   INSTALL="sh 
> ${.CURDIR}/tools/install.sh" \
> /usr/main-src/Makefile.inc1:KTMAKEENV=  INSTALL="sh 
> ${.CURDIR}/tools/install.sh" \
> 
> Also:
> 
> # kernel-tools stage
> KTMAKEENV=  INSTALL="sh ${.CURDIR}/tools/install.sh" \
> 
> vs.
> 
> # world stage
> WMAKEENV=   ${CROSSENV} \
>INSTALL="${INSTALL_CMD} -U" \
> and:
> .if defined(DB_FROM_SRC) || defined(NO_ROOT)
> IMAKE_INSTALL=  INSTALL="${INSTALL_CMD} ${INSTALLFLAGS}"
> 
> So: explicitly varying styles for various contexts.
> 
> It looks to me like the make output would show:
> 
> install . . . 
> vs.
> sh . . ./tools/install.sh . . .
> 
> based on if the script is not-in-use