On Apr 4, 2025, at 22:50, Mark Millard <mark...@yahoo.com> wrote: > Philip Paeps <philip_at_freebsd.org> wrote on > Date: Sat, 05 Apr 2025 03:10:26 UTC : > >> On 2025-04-05 00:37:47 (+0800), Kevin Oberman wrote: >>> There appear to have been no successful package updates for FreeBSD >>> since >>> EOL for 14.1-AMD64. The only build for 142amd64-default that one >>> appears to >>> have crashed. As a result, I am seeing a growing list of packages that >>> are >>> not getting updated. The last successful build was the final build on >>> 14.1 >>> last Monday March 30. >>> >>> Any idea how long it might be or how serious the issue is? It looks >>> like >>> all package builds were completed, though there does appear to have >>> been a >>> significant jump in failed builds since the previous build (14.1 >>> default) >> >> We upgraded the package builders to recent -CURRENT last week and the >> release/X build jails are no longer working. > > Well: The i386 builds do not crash (reach "done:"). The amd64 > ones do crash. > > Separate issue: > pkg 2.1.0 also makes the overall build sequence take much longer, > especially on older/slower hardware. For example, beefy17 and > beefy18 now take over twice as long. Even beefy21 and beefy22 > take over 9 more hrs, over 20% longer. > >> If we have to reinstall >> the builders with older -CURRENT, it's at least a week of work. If >> someone can fix the actual bug in -CURRENT, it'll take us about an >> hour-ish to upgrade all the builders. >> >> I don't know what the bug is. But it's easy enough to reproduce: run >> poudriere in a release/X jail on -CURRENT and watch it explode. > > That form of testing appears to presume i386 is not the type > of context. > > For amd64, it is not obvious to me for: > > Host OSVERSION: 1500035 (prior: 1500028) > vs. > Jail OSVERSION: 1402000 (prior: 1401000) > > for what might be the important part, although I'd tend to > guess the Host 1500035 part. (Both parts were changed at > the same time.)
So I upgraded my 1500034 PkgBased zfs environment to have a modern 1500035 PkgBase kernel set, leaving the world alone and the ports alone ( predating pkg 2.1.0 ). I selected to boot the debug PkgBase kernel. So . . . # uname -apKU FreeBSD 7950X3D-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT main-n276250-b1c62081feec GENERIC amd64 amd64 1500035 1500034 # poudriere jail -l JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP PATH release-amd64 14.2-RELEASE-p2 amd64 pkgbase 2025-03-12 21:09:52 /usr/local/poudriere/jails/release-amd64 . . . # poudriere bulk -jrelease-amd64 -C ports-mgmt/pkg [00:00:00] Creating the reference jail... done . . . [00:00:27] [01] [00:00:25] Finished ports-mgmt/pkg | pkg-2.0.6: Success . . . [00:00:28] Logs: /usr/local/poudriere/data/logs/bulk/release-amd64-default/2025-04-04_23h41m04s [00:00:28] Cleaning up [00:00:28] Unmounting file systems No crash occurred. For reference: =>> Building ports-mgmt/pkg build started at 2025-04-04T23:41:06-07:00 port directory: /usr/ports/ports-mgmt/pkg package name: pkg-2.0.6 building for: FreeBSD ZNV4optb_ZFS 14.2-RELEASE-p2 FreeBSD 14.2-RELEASE-p2 amd64 maintained by: p...@freebsd.org port version: 2.0.6 port revision: 0 Makefile datestamp: -rw-r--r-- 1 root wheel 1422 Feb 12 22:03 /usr/ports/ports-mgmt/pkg/Makefile Ports top last git commit: 87a21eb4786a Ports top unclean checkout: yes =>> Inspecting /usr/local/poudriere/data/.m/release-amd64-default/01//usr/ports/ports-mgmt/pkg for modifications to git checkout... no Port dir last git commit: 3fc159574db3 Port dir unclean checkout: no Poudriere version: poudriere-git-3.4.99.20250209 Host OSVERSION: 1500035 Jail OSVERSION: 1402000 Job Id: 01 . . . But there was also: lock order reversal: 1st 0xfffff801c5b69070 zfs (zfs, lockmgr) @ /home/pkgbuild/worktrees/main/sys/kern/vfs_mount.c:2260 2nd 0xfffff802bf461ac0 tmpfs (tmpfs, lockmgr) @ /home/pkgbuild/worktrees/main/sys/kern/vfs_subr.c:4178 lock order tmpfs -> zfs established at: #0 0xffffffff80bce334 at witness_checkorder+0x364 #1 0xffffffff80b254e1 at lockmgr_xlock+0x51 #2 0xffffffff83569d18 at null_lock+0xb8 #3 0xffffffff80c6d5e3 at _vn_lock+0x53 #4 0xffffffff80c55851 at vflush+0x101 #5 0xffffffff83568c50 at nullfs_unmount+0x50 #6 0xffffffff80c4a1b9 at dounmount+0x649 #7 0xffffffff80c49b0c at kern_unmount+0x2dc #8 0xffffffff81097a0a at amd64_syscall+0x15a #9 0xffffffff8106926b at fast_syscall_common+0xf8 lock order zfs -> tmpfs attempted at: #0 0xffffffff80bcebb1 at witness_checkorder+0xbe1 #1 0xffffffff80b23c0a at lockmgr_lock_flags+0x16a #2 0xffffffff80c6d5e3 at _vn_lock+0x53 #3 0xffffffff80c55851 at vflush+0x101 #4 0xffffffff80a7b230 at tmpfs_unmount+0x70 #5 0xffffffff80c4a1b9 at dounmount+0x649 #6 0xffffffff80c49b0c at kern_unmount+0x2dc #7 0xffffffff81097a0a at amd64_syscall+0x15a #8 0xffffffff8106926b at fast_syscall_common+0xf8 I do not know if that is likely to be relevant. I've no clue what all to use to closely match what the official builders use for poudriere.conf content, such as for USE_TMPFS=??? . So far I'm not seeing any evidence that: Host OSVERSION: 1500035 Jail OSVERSION: 1402000 is, of itself, a problem. === Mark Millard marklmi at yahoo.com
