[142amd64-quarterly's incremental build that started on "03 Apr 2025 01:01:38 GMT" on beefy20 has finish, reaching "done:" Status.]
On Apr 5, 2025, at 00:05, Mark Millard <mark...@yahoo.com> wrote: >> On Apr 4, 2025, at 22:50, Mark Millard <mark...@yahoo.com> wrote: >> >>> Philip Paeps <philip_at_freebsd.org> wrote on >>> Date: Sat, 05 Apr 2025 03:10:26 UTC : >>> >>>> On 2025-04-05 00:37:47 (+0800), Kevin Oberman wrote: >>>>> There appear to have been no successful package updates for FreeBSD >>>>> since >>>>> EOL for 14.1-AMD64. The only build for 142amd64-default that one >>>>> appears to >>>>> have crashed. As a result, I am seeing a growing list of packages that >>>>> are >>>>> not getting updated. The last successful build was the final build on >>>>> 14.1 >>>>> last Monday March 30. >>>>> >>>>> Any idea how long it might be or how serious the issue is? It looks >>>>> like >>>>> all package builds were completed, though there does appear to have >>>>> been a >>>>> significant jump in failed builds since the previous build (14.1 >>>>> default) >>>> >>>> We upgraded the package builders to recent -CURRENT last week and the >>>> release/X build jails are no longer working. >>> >>> Well: The i386 builds do not crash (reach "done:"). The amd64 >>> ones do crash. >>> >>> Separate issue: >>> pkg 2.1.0 also makes the overall build sequence take much longer, >>> especially on older/slower hardware. For example, beefy17 and >>> beefy18 now take over twice as long. Even beefy21 and beefy22 >>> take over 9 more hrs, over 20% longer. >>> >>>> If we have to reinstall >>>> the builders with older -CURRENT, it's at least a week of work. If >>>> someone can fix the actual bug in -CURRENT, it'll take us about an >>>> hour-ish to upgrade all the builders. >>>> >>>> I don't know what the bug is. But it's easy enough to reproduce: run >>>> poudriere in a release/X jail on -CURRENT and watch it explode. >>> >>> That form of testing appears to presume i386 is not the type >>> of context. >>> >>> For amd64, it is not obvious to me for: >>> >>> Host OSVERSION: 1500035 (prior: 1500028) >>> vs. >>> Jail OSVERSION: 1402000 (prior: 1401000) >>> >>> for what might be the important part, although I'd tend to >>> guess the Host 1500035 part. (Both parts were changed at >>> the same time.) >> >> So I upgraded my 1500034 PkgBased zfs environment to have a modern >> 1500035 PkgBase kernel set, leaving the world alone and the ports >> alone ( predating pkg 2.1.0 ). I selected to boot the debug >> PkgBase kernel. So . . . >> >> # uname -apKU >> FreeBSD 7950X3D-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT >> main-n276250-b1c62081feec GENERIC amd64 amd64 1500035 1500034 >> >> # poudriere jail -l >> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP >> PATH >> release-amd64 14.2-RELEASE-p2 amd64 pkgbase 2025-03-12 21:09:52 >> /usr/local/poudriere/jails/release-amd64 >> . . . >> >> # poudriere bulk -jrelease-amd64 -C ports-mgmt/pkg >> [00:00:00] Creating the reference jail... done >> . . . >> [00:00:27] [01] [00:00:25] Finished ports-mgmt/pkg | pkg-2.0.6: Success >> . . . >> [00:00:28] Logs: >> /usr/local/poudriere/data/logs/bulk/release-amd64-default/2025-04-04_23h41m04s >> [00:00:28] Cleaning up >> [00:00:28] Unmounting file systems >> >> No crash occurred. >> >> For reference: >> >> =>> Building ports-mgmt/pkg >> build started at 2025-04-04T23:41:06-07:00 >> port directory: /usr/ports/ports-mgmt/pkg >> package name: pkg-2.0.6 >> building for: FreeBSD ZNV4optb_ZFS 14.2-RELEASE-p2 FreeBSD 14.2-RELEASE-p2 >> amd64 >> maintained by: p...@freebsd.org >> port version: 2.0.6 >> port revision: 0 >> Makefile datestamp: -rw-r--r-- 1 root wheel 1422 Feb 12 22:03 >> /usr/ports/ports-mgmt/pkg/Makefile >> Ports top last git commit: 87a21eb4786a >> Ports top unclean checkout: yes >> =>> Inspecting >> /usr/local/poudriere/data/.m/release-amd64-default/01//usr/ports/ports-mgmt/pkg >> for modifications to git checkout... no >> Port dir last git commit: 3fc159574db3 >> Port dir unclean checkout: no >> Poudriere version: poudriere-git-3.4.99.20250209 >> Host OSVERSION: 1500035 >> Jail OSVERSION: 1402000 >> Job Id: 01 >> . . . >> >> >> But there was also: >> >> lock order reversal: >> 1st 0xfffff801c5b69070 zfs (zfs, lockmgr) @ >> /home/pkgbuild/worktrees/main/sys/kern/vfs_mount.c:2260 >> 2nd 0xfffff802bf461ac0 tmpfs (tmpfs, lockmgr) @ >> /home/pkgbuild/worktrees/main/sys/kern/vfs_subr.c:4178 >> lock order tmpfs -> zfs established at: >> #0 0xffffffff80bce334 at witness_checkorder+0x364 >> #1 0xffffffff80b254e1 at lockmgr_xlock+0x51 >> #2 0xffffffff83569d18 at null_lock+0xb8 >> #3 0xffffffff80c6d5e3 at _vn_lock+0x53 >> #4 0xffffffff80c55851 at vflush+0x101 >> #5 0xffffffff83568c50 at nullfs_unmount+0x50 >> #6 0xffffffff80c4a1b9 at dounmount+0x649 >> #7 0xffffffff80c49b0c at kern_unmount+0x2dc >> #8 0xffffffff81097a0a at amd64_syscall+0x15a >> #9 0xffffffff8106926b at fast_syscall_common+0xf8 >> lock order zfs -> tmpfs attempted at: >> #0 0xffffffff80bcebb1 at witness_checkorder+0xbe1 >> #1 0xffffffff80b23c0a at lockmgr_lock_flags+0x16a >> #2 0xffffffff80c6d5e3 at _vn_lock+0x53 >> #3 0xffffffff80c55851 at vflush+0x101 >> #4 0xffffffff80a7b230 at tmpfs_unmount+0x70 >> #5 0xffffffff80c4a1b9 at dounmount+0x649 >> #6 0xffffffff80c49b0c at kern_unmount+0x2dc >> #7 0xffffffff81097a0a at amd64_syscall+0x15a >> #8 0xffffffff8106926b at fast_syscall_common+0xf8 >> >> >> I do not know if that is likely to be relevant. >> >> I've no clue what all to use to closely match what >> the official builders use for poudriere.conf >> content, such as for USE_TMPFS=??? . >> >> So far I'm not seeing any evidence that: >> >> Host OSVERSION: 1500035 >> Jail OSVERSION: 1402000 >> >> is, of itself, a problem. > > 142amd64-quarterly 359bbf7fc5af on beefy20 finished with "done:" status: No crash. It built 26082 of 27764 queued packages. So it was an incremental bulk build. It still has a large Skipped figure (1286). Some major contributions were the failures of: go121-1.21.13_5 (skipped 542) webkit2-gtk_40-2.46.6 (skipped 323) go123-1.23.7_4 (skipped 100) It appears the earlier 142amd64-quarterly 20ba05c2c778 type of crash is not guaranteed to happen. (My earlier, simple attempts at reproduction also got no crash.) 142amd64-default 4ff0a7ba4d1f on beefy22 is in process and is another incremental bulk build. Side note about pkg 2.1.0 being in use: I'll note that beefy20 showed 7 lib-depends active and each had over 11 min of elapsed time showing: (beefy20 is 1 of the 3 fastest builder systems) 09 libkgapi-24.12.3 net/libkgapi lib-depends 00:12:58 20 plasma6-libksysguard-6.3.4 sysutils/plasma6-libksysguard lib-depends 00:12:39 05 konqueror-24.12.3 x11-fm/konqueror lib-depends 00:12:12 18 khelpcenter-24.12.3 sysutils/khelpcenter lib-depends 00:12:02 11 parley-24.12.3 misc/parley lib-depends 00:11:51 37 cantor-24.12.3_1 math/cantor lib-depends 00:11:45 41 marble-24.12.3 astro/marble lib-depends 00:11:42 === Mark Millard marklmi at yahoo.com
