[Just fixing a beffy20 reference to be beefy22.] On Apr 5, 2025, at 09:53, Mark Millard <mark...@yahoo.com> wrote:
> [142amd64-quarterly's incremental build that started on > "03 Apr 2025 01:01:38 GMT" on beefy20 has finish, reaching > "done:" Status.] > > On Apr 5, 2025, at 00:05, Mark Millard <mark...@yahoo.com> wrote: > >>> On Apr 4, 2025, at 22:50, Mark Millard <mark...@yahoo.com> wrote: >>> >>>> Philip Paeps <philip_at_freebsd.org> wrote on >>>> Date: Sat, 05 Apr 2025 03:10:26 UTC : >>>> >>>>> On 2025-04-05 00:37:47 (+0800), Kevin Oberman wrote: >>>>>> There appear to have been no successful package updates for FreeBSD >>>>>> since >>>>>> EOL for 14.1-AMD64. The only build for 142amd64-default that one >>>>>> appears to >>>>>> have crashed. As a result, I am seeing a growing list of packages that >>>>>> are >>>>>> not getting updated. The last successful build was the final build on >>>>>> 14.1 >>>>>> last Monday March 30. >>>>>> >>>>>> Any idea how long it might be or how serious the issue is? It looks >>>>>> like >>>>>> all package builds were completed, though there does appear to have >>>>>> been a >>>>>> significant jump in failed builds since the previous build (14.1 >>>>>> default) >>>>> >>>>> We upgraded the package builders to recent -CURRENT last week and the >>>>> release/X build jails are no longer working. >>>> >>>> Well: The i386 builds do not crash (reach "done:"). The amd64 >>>> ones do crash. >>>> >>>> Separate issue: >>>> pkg 2.1.0 also makes the overall build sequence take much longer, >>>> especially on older/slower hardware. For example, beefy17 and >>>> beefy18 now take over twice as long. Even beefy21 and beefy22 >>>> take over 9 more hrs, over 20% longer. >>>> >>>>> If we have to reinstall >>>>> the builders with older -CURRENT, it's at least a week of work. If >>>>> someone can fix the actual bug in -CURRENT, it'll take us about an >>>>> hour-ish to upgrade all the builders. >>>>> >>>>> I don't know what the bug is. But it's easy enough to reproduce: run >>>>> poudriere in a release/X jail on -CURRENT and watch it explode. >>>> >>>> That form of testing appears to presume i386 is not the type >>>> of context. >>>> >>>> For amd64, it is not obvious to me for: >>>> >>>> Host OSVERSION: 1500035 (prior: 1500028) >>>> vs. >>>> Jail OSVERSION: 1402000 (prior: 1401000) >>>> >>>> for what might be the important part, although I'd tend to >>>> guess the Host 1500035 part. (Both parts were changed at >>>> the same time.) >>> >>> So I upgraded my 1500034 PkgBased zfs environment to have a modern >>> 1500035 PkgBase kernel set, leaving the world alone and the ports >>> alone ( predating pkg 2.1.0 ). I selected to boot the debug >>> PkgBase kernel. So . . . >>> >>> # uname -apKU >>> FreeBSD 7950X3D-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT >>> main-n276250-b1c62081feec GENERIC amd64 amd64 1500035 1500034 >>> >>> # poudriere jail -l >>> JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP >>> PATH >>> release-amd64 14.2-RELEASE-p2 amd64 pkgbase 2025-03-12 >>> 21:09:52 /usr/local/poudriere/jails/release-amd64 >>> . . . >>> >>> # poudriere bulk -jrelease-amd64 -C ports-mgmt/pkg >>> [00:00:00] Creating the reference jail... done >>> . . . >>> [00:00:27] [01] [00:00:25] Finished ports-mgmt/pkg | pkg-2.0.6: Success >>> . . . >>> [00:00:28] Logs: >>> /usr/local/poudriere/data/logs/bulk/release-amd64-default/2025-04-04_23h41m04s >>> [00:00:28] Cleaning up >>> [00:00:28] Unmounting file systems >>> >>> No crash occurred. >>> >>> For reference: >>> >>> =>> Building ports-mgmt/pkg >>> build started at 2025-04-04T23:41:06-07:00 >>> port directory: /usr/ports/ports-mgmt/pkg >>> package name: pkg-2.0.6 >>> building for: FreeBSD ZNV4optb_ZFS 14.2-RELEASE-p2 FreeBSD 14.2-RELEASE-p2 >>> amd64 >>> maintained by: p...@freebsd.org >>> port version: 2.0.6 >>> port revision: 0 >>> Makefile datestamp: -rw-r--r-- 1 root wheel 1422 Feb 12 22:03 >>> /usr/ports/ports-mgmt/pkg/Makefile >>> Ports top last git commit: 87a21eb4786a >>> Ports top unclean checkout: yes >>> =>> Inspecting >>> /usr/local/poudriere/data/.m/release-amd64-default/01//usr/ports/ports-mgmt/pkg >>> for modifications to git checkout... no >>> Port dir last git commit: 3fc159574db3 >>> Port dir unclean checkout: no >>> Poudriere version: poudriere-git-3.4.99.20250209 >>> Host OSVERSION: 1500035 >>> Jail OSVERSION: 1402000 >>> Job Id: 01 >>> . . . >>> >>> >>> But there was also: >>> >>> lock order reversal: >>> 1st 0xfffff801c5b69070 zfs (zfs, lockmgr) @ >>> /home/pkgbuild/worktrees/main/sys/kern/vfs_mount.c:2260 >>> 2nd 0xfffff802bf461ac0 tmpfs (tmpfs, lockmgr) @ >>> /home/pkgbuild/worktrees/main/sys/kern/vfs_subr.c:4178 >>> lock order tmpfs -> zfs established at: >>> #0 0xffffffff80bce334 at witness_checkorder+0x364 >>> #1 0xffffffff80b254e1 at lockmgr_xlock+0x51 >>> #2 0xffffffff83569d18 at null_lock+0xb8 >>> #3 0xffffffff80c6d5e3 at _vn_lock+0x53 >>> #4 0xffffffff80c55851 at vflush+0x101 >>> #5 0xffffffff83568c50 at nullfs_unmount+0x50 >>> #6 0xffffffff80c4a1b9 at dounmount+0x649 >>> #7 0xffffffff80c49b0c at kern_unmount+0x2dc >>> #8 0xffffffff81097a0a at amd64_syscall+0x15a >>> #9 0xffffffff8106926b at fast_syscall_common+0xf8 >>> lock order zfs -> tmpfs attempted at: >>> #0 0xffffffff80bcebb1 at witness_checkorder+0xbe1 >>> #1 0xffffffff80b23c0a at lockmgr_lock_flags+0x16a >>> #2 0xffffffff80c6d5e3 at _vn_lock+0x53 >>> #3 0xffffffff80c55851 at vflush+0x101 >>> #4 0xffffffff80a7b230 at tmpfs_unmount+0x70 >>> #5 0xffffffff80c4a1b9 at dounmount+0x649 >>> #6 0xffffffff80c49b0c at kern_unmount+0x2dc >>> #7 0xffffffff81097a0a at amd64_syscall+0x15a >>> #8 0xffffffff8106926b at fast_syscall_common+0xf8 >>> >>> >>> I do not know if that is likely to be relevant. >>> >>> I've no clue what all to use to closely match what >>> the official builders use for poudriere.conf >>> content, such as for USE_TMPFS=??? . >>> >>> So far I'm not seeing any evidence that: >>> >>> Host OSVERSION: 1500035 >>> Jail OSVERSION: 1402000 >>> >>> is, of itself, a problem. >> >> > > 142amd64-quarterly 359bbf7fc5af on beefy20 finished with > "done:" status: No crash. > > It built 26082 of 27764 queued packages. So it was an > incremental bulk build. > > It still has a large Skipped figure (1286). Some major > contributions were the failures of: > > go121-1.21.13_5 (skipped 542) > webkit2-gtk_40-2.46.6 (skipped 323) > go123-1.23.7_4 (skipped 100) > > It appears the earlier 142amd64-quarterly 20ba05c2c778 > type of crash is not guaranteed to happen. (My earlier, > simple attempts at reproduction also got no crash.) > > > 142amd64-default 4ff0a7ba4d1f on beefy22 is in process > and is another incremental bulk build. > > > Side note about pkg 2.1.0 being in use: > > I'll note that beefy20 make that: beefy22 (the one still in process) > showed 7 lib-depends active > and each had over 11 min of elapsed time showing: > (beefy20 is 1 of the 3 fastest builder systems) > > 09 libkgapi-24.12.3 net/libkgapi lib-depends 00:12:58 > 20 plasma6-libksysguard-6.3.4 sysutils/plasma6-libksysguard lib-depends > 00:12:39 > 05 konqueror-24.12.3 x11-fm/konqueror lib-depends 00:12:12 > 18 khelpcenter-24.12.3 sysutils/khelpcenter lib-depends 00:12:02 > 11 parley-24.12.3 misc/parley lib-depends 00:11:51 > 37 cantor-24.12.3_1 math/cantor lib-depends 00:11:45 > 41 marble-24.12.3 astro/marble lib-depends 00:11:42 === Mark Millard marklmi at yahoo.com
