On 2021/11/13 23:05, Stuart Henderson wrote: > On 2021/11/13 22:41, Stuart Henderson wrote: > > On 2021/11/13 21:49, Stuart Henderson wrote: > > > On 2021/11/13 18:04, Alexander Bluhm wrote: > > > > Hi, > > > > > > > > To make IPsec MP safe we need refcounting for the tdb. The diff > > > > below is part of something bigger we have at genua. Although it > > > > does not cover timeouts and the tdb reaper yet, I want to get this > > > > in as a frist step. > > > > > > > > It passes regress but there are setups that are not covered. Bridge > > > > and pfsync with IPsec and TCP signature need special care. > > > > > > > > When testing, please check for tdb leaks. The vmstat -m tdb in use > > > > culumn must be 0. It it is not, try ipsecctl -F. If ipsecctl -sa > > > > shows nothing, but vmstat -m shows a used tdb, then it is a bug. > > > > > > > > Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg > > > > Maxpg Idle > > > > tdb 904 56 0 0 7 7 0 7 0 > > > > 8 0 > > > > > > Tested with a box running isakmpd at home. Config is: > > > > > > isakmpd_flags=-Kv -D0=29 -D1=49 -D2=10 -D3=30 -D5=20 -D6=30 -D8=30 -D9=30 > > > -D10=20 > > > > > > ike passive esp \ > > > from xxx.xx.xxx.1 (xxx.xx.xxx.0/27) to 0.0.0.0/0 \ > > > local xxx.xx.xxx.1 \ > > > main auth hmac-sha2-256 enc aes group ecp256 \ > > > quick enc aes-128-gcm group ecp256 \ > > > srcid xxx > > > > > > isakmpd starts at boot, 3 devices connect in and tunnels come up, > > > FLOWS and SAD show in ipsecctl -sa. > > > > > > isakmpd crashes shortly after starting (as described in previous mails; > > > this is not new, it happens very often on the first start at boot), > > > FLOWS and SAD remain showing. At this point they still show InUse: > > > > > > Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg > > > Maxpg Idle > > > tdb 1088 9 0 6 1 0 1 1 0 > > > 8 0 > > > > > > After ipsecctl -F they are freed: > > > > > > tdb 1088 9 0 0 1 0 1 1 0 > > > 8 1 > > > > > > Restart isakmpd, ipsecctl -f /etc/ipsec.conf, devices connect back in: > > > > > > tdb 1088 18 0 6 2 1 1 1 0 > > > 8 0 > > > > > > (And at this point isakmpd is usually stable). > > > > > > I will leave a shell loop running vmstat -m every few minutes to check > > > if it grows. > > > > > > (Bad timing for testing TCPMD5 because over the last couple of days I have > > > just removed bgpd+ospfd from the machines that would have been good > > > choices to run test diffs..I might be able to get some simple setup > > > back up and running though) > > > > Ah, that was interesting: > > > > Sat Nov 13 21:44:17 GMT 2021 > > tdb 1088 18 0 6 2 1 1 1 0 8 > > 0 > > > > Sat Nov 13 21:54:44 GMT 2021 > > tdb 1088 18 0 6 2 1 1 1 0 8 > > 0 > > > > Sat Nov 13 22:04:44 GMT 2021 > > tdb 1088 27 0 12 2 1 1 1 0 8 > > 0 > > > > and then shortly after, boom. Unfortunately I didn't get > > much from ddb before it hanged after I tried changing cpu. > > > > [playback] > > Xsyscall() at Xsyscall+0x128 > > end of kernel > > end trace frame: 0x7f7ffffb2d20, count: 248 > > End of stack trace. > > splassert: tdb_free: want 2 have 0 > > Starting stack trace... > > tdb_free(ffff800000f40b88) at tdb_free+0x47 > > pfkeyv2_send(fffffd80603603b8,ffff800000f44c00,170) at pfkeyv2_send+0x656 > > pfkeyv2_output(fffffd8064cb5e00,fffffd80603603b8,0,0) at pfkeyv2_output+0x8a > > pfkeyv2_usrreq(fffffd80603603b8,9,fffffd8064cb5e00,0,0,ffff8000ffff9508) at > > pfkeyv2_usrreq+0x1b0 > > sosend(fffffd80603603b8,0,ffff80001fd8f6f0,0,0,0) at sosend+0x3a9 > > > > dofilewritev(ffff8000ffff9508,7,ffff80001fd8f6f0,0,ffff80001fd8f7f0) at > > dofilewritev+0x14d > > sys_writev(ffff8000ffff9508,ffff80001fd8f790,ffff80001fd8f7f0) at > > sys_writev+0xd2 > > syscall(ffff80001fd8f860) at syscall+0x3a9 > > > > Xsyscall() at Xsyscall+0x128 > > end of kernel > > end trace frame: 0x7f7ffffb2d20, count: 248 > > End of stack trace. > > splassert: tdb_free: want 2 have 0 > > Starting stack trace... > > tdb_free(ffff800000f40308) at tdb_free+0x47 > > pfkeyv2_send(fffffd80603603b8,ffff800000eff300,50) at pfkeyv2_send+0x656 > > pfkeyv2_output(fffffd8062506e00,fffffd80603603b8,0,0) at pfkeyv2_output+0x8a > > pfkeyv2_usrreq(fffffd80603603b8,9,fffffd8062506e00,0,0,ffff8000ffff9508) at > > pfkeyv2_usrreq+0x1b0 > > sosend(fffffd80603603b8,0,ffff80001fd8f6f0,0,0,0) at sosend+0x3a9 > > > > dofilewritev(ffff8000ffff9508,7,ffff80001fd8f6f0,0,ffff80001fd8f7f0) at > > dofilewritev+0x14d > > sys_writev(ffff8000ffff9508,ffff80001fd8f790,ffff80001fd8f7f0) at > > sys_writev+0xd2 > > syscall(ffff80001fd8f860) at syscall+0x3a9 > > > > Xsyscall() at Xsyscall+0x128 > > end of kernel > > end trace frame: 0x7f7ffffc3170, count: 248 > > End of stack trace. > > splassert: ipsp_ids_free: want 2 have 0 > > Starting stack trace... > > ipsp_ids_free(ffff800000f30880) at ipsp_ids_free+0x4e > > tdb_free(ffff800000f40308) at tdb_free+0x13c > > pfkeyv2_send(fffffd80603603b8,ffff800000eff300,50) at pfkeyv2_send+0x656 > > pfkeyv2_output(fffffd8062506e00,fffffd80603603b8,0,0) at pfkeyv2_output+0x8a > > pfkeyv2_usrreq(fffffd80603603b8,9,fffffd8062506e00,0,0,ffff8000ffff9508) at > > pfkeyv2_usrreq+0x1b0 > > sosend(fffffd80603603b8,0,ffff80001fd8f6f0,0,0,0) at sosend+0x3a9 > > > > dopfainlice:wr iketernv(eflf ffd8i0a0gn0offsftfic9 5a08ss,7er,ftfiofnf > > 8"0r00ef1cfdnt8 f6!=f 0,~00",f ffaiflfe80d:0 01fiflde8 f7"f/s0ys)/ akter > > n/kern_ > > sdyofncilh.ecw"ri,t elivn+e0 x8124d4 > > > > Stopped at db_enter+0x10: popq %rbp > > TID PID UID PRFLAGS PFLAGS CPU COMMAND > > 185954 5669 68 0x10 0 1 isakmpd > > 67015 91815 0 0x14000 0x200 3 softnet > > *318814 6181 0 0x14000 0x40000200 0 softclock > > db_enter() at db_enter+0x10 > > panic(ffffffff81e4a37b) at panic+0xbf > > __assert(ffffffff81eb3b74,ffffffff81e1e19f,338,ffffffff81e4f078) at > > __assert+0x > > 25 > > > > refcnt_rele(ffff800000f40330) at refcnt_rele+0x6f > > tdb_delete(ffff800000f40308) at tdb_delete+0x84 > > tdb_timeout(ffff800000f40308) at tdb_timeout+0x39 > > softclock_thread(ffff8000fffff500) at softclock_thread+0x16e > > end trace frame: 0x0, count: 8 > > https://www.openbsd.org/ddb.html describes the minimum info required in bug > > reports. Insufficient info makes it difficult to find and fix bugs. > > ddb{0}> [-- sthen@localhost attached -- Sat Nov 13 22:23:17 2021] > > tr > > db_enter() at db_enter+0x10 > > panic(ffffffff81e4a37b) at panic+0xbf > > __assert(ffffffff81eb3b74,ffffffff81e1e19f,338,ffffffff81e4f078) at > > __assert+0x > > 25 > > > > refcnt_rele(ffff800000f40330) at refcnt_rele+0x6f > > tdb_delete(ffff800000f40308) at tdb_delete+0x84 > > tdb_timeout(ffff800000f40308) at tdb_timeout+0x39 > > softclock_thread(ffff8000fffff500) at softclock_thread+0x16e > > end trace frame: 0x0, count: -7 > > ddb{0}> sh reg > > rdi 0 > > rsi 0x14 > > rbp 0xffff80001fbdc6e0 > > rbx 0 > > rdx 0xde00000003ff0239 > > rcx 0x202 > > rax 0x62 > > r8 0x101010101010101 > > r9 0 > > r10 0 > > r11 0x254de177953953f9 > > r12 0xffffffff82196a00 cpu_info_full_primary+0x2a00 > > r13 0xffffffff82275110 timeout_proc > > r14 0 > > r15 0xffffffff81e4a37b cmd0646_9_tim_udma+0x2e00b > > rip 0xffffffff81dad9f0 db_enter+0x10 > > cs 0x8 > > rflags 0x206 > > rsp 0xffff80001fbdc6e0 > > ss 0x10 > > db_enter+0x10: popq %rbp > > ddb{0}> ps > > PID TID PPID UID S FLAGS WAIT COMMAND > > 20274 32066 93874 1000 3 0x100083 nanoslp sleep > > 93874 445500 10192 1000 3 0x10008b sigsusp ksh > > 10192 370220 36870 1000 3 0x98 kqread sshd > > 36870 222408 98260 0 3 0x92 kqread sshd > > 5669 185954 75457 68 7 0x10 isakmpd > > 75457 276857 1 0 3 0x80 netio isakmpd > > 21598 237571 1 0 3 0x100083 ttyin getty > > 4506 488964 1 0 3 0x100098 kqread cron > > 39797 52192 15626 846 3 0x100098 kqread dhcpcd > > 59318 229873 15626 846 3 0x100098 kqread dhcpcd > > 7830 517562 15626 0 3 0x88 kqread dhcpcd > > 15626 323417 1 846 3 0x100098 kqread dhcpcd > > 19237 354049 75741 720 3 0x90 kqread lldpd > > 75741 66119 1 0 3 0x80 netio lldpd > > 61311 177515 1 99 3 0x100090 kqread sndiod > > 11783 282500 1 110 3 0x100090 kqread sndiod > > 82651 136045 37762 0 3 0x100090 kqread tftp-proxy > > 37762 379684 1 108 3 0x90 kqread tftp-proxy > > 1745 324688 1 109 3 0x90 kqread ftp-proxy > > 39681 116848 88999 95 3 0x100092 kqread smtpd > > 25384 256665 88999 103 3 0x100092 kqread smtpd > > 92345 176419 88999 95 3 0x100092 kqread smtpd > > 55442 232201 88999 95 3 0x100092 kqread smtpd > > 77443 215878 88999 95 3 0x100092 kqread smtpd > > 69162 261319 88999 95 3 0x100092 kqread smtpd > > 88999 111572 1 0 3 0x100080 kqread smtpd > > 58046 516005 58609 94 3 0x100092 kqread rad > > 90680 269647 58609 94 3 0x100092 kqread rad > > 58609 188614 1 0 3 0x100080 kqread rad > > 3419 227743 1 77 3 0x100090 kqread dhcpd > > 61767 205183 1 0 3 0x100080 kqread snmpd > > 51987 257332 1 91 3 0x92 kqread snmpd > > 98260 56979 1 0 3 0x88 kqread sshd > > 13528 58314 1 0 3 0x100080 kqread ntpd > > 18801 320473 51204 83 3 0x100092 kqread ntpd > > 51204 405642 1 83 3 0x100092 kqread ntpd > > 53483 209302 30989 53 3 0x90 kqread unbound > > 30989 499872 1 53 3 0x90 kqread unbound > > 16457 234403 32406 74 3 0x100092 bpf pflogd > > 32406 88142 1 0 3 0x80 netio pflogd > > 80238 516045 79768 73 3 0x100090 kqread syslogd > > 79768 67472 1 0 3 0x100082 netio syslogd > > 43550 409362 1 0 3 0x100080 kqread resolvd > > 442 379277 99464 77 3 0x100092 kqread dhcpleased > > 47451 452762 99464 77 3 0x100092 kqread dhcpleased > > 99464 433528 1 0 3 0x80 kqread dhcpleased > > 21886 318205 0 0 3 0x14200 bored wg_crypt > > 97050 229388 0 0 3 0x14200 bored wg_crypt > > 4440 5494 0 0 3 0x14200 bored wg_crypt > > 72609 120590 0 0 3 0x14200 bored wg_crypt > > 20025 29295 0 0 3 0x14200 bored wg_handshake > > 43757 478463 0 0 3 0x14200 bored wg_handshake > > 55892 346923 51675 115 3 0x100092 kqread slaacd > > 76843 163407 51675 115 3 0x100092 kqread slaacd > > 51675 404409 1 0 3 0x100080 kqread slaacd > > 28362 202426 0 0 3 0x14200 bored smr > > 82207 376534 0 0 3 0x14200 pgzero zerothread > > 1524 51196 0 0 3 0x14200 aiodoned aiodoned > > 22770 341109 0 0 3 0x14200 syncer update > > 33241 337755 0 0 3 0x14200 cleaner cleaner > > 27893 102676 0 0 3 0x14200 reaper reaper > > 79368 439400 0 0 3 0x14200 pgdaemon pagedaemon > > 98877 12806 0 0 3 0x14200 bored sensors > > 71274 388091 0 0 3 0x14200 mmctsk sdmmc0 > > 33311 377928 0 0 3 0x14200 usbtsk usbtask > > 83240 116348 0 0 3 0x14200 usbatsk usbatsk > > 45461 344150 0 0 3 0x40014200 acpi0 acpi0 > > 94708 135825 0 0 3 0x40014200 idle3 > > 46611 496404 0 0 7 0x40014200 idle2 > > 26280 26936 0 0 3 0x40014200 idle1 > > 91815 67015 0 0 7 0x14200 softnet > > 90110 476121 0 0 3 0x14200 bored systqmp > > 12475 134606 0 0 3 0x14200 bored systq > > * 6181 318814 0 0 7 0x40014200 softclock > > 38662 5703 0 0 3 0x40014200 idle0 > > 1 335266 0 0 3 0x82 wait init > > 0 0 -1 0 3 0x10200 scheduler swapper > > ddb{0}> ps /o > > TID PID UID PRFLAGS PFLAGS CPU COMMAND > > 185954 5669 68 0x10 0 1 isakmpd > > 67015 91815 0 0x14000 0x200 3 softnet > > *318814 6181 0 0x14000 0x40000200 0 softclock > > ddb{0}> tr /t 0t185954 > > ffff800000f40308(ffffffffffffffff,ffffffffffffffff,ffff800000f402e8,0,0,0) > > at 0 > > xffff800000f40308 > > > > end trace frame: 0x0, count: -1 > > ddb{0}> mach ddbcpu 1 > > > > <ddb hanged here> > > > > However, looking at the earlier console log, there were splasserts > > starting from before it went multiuser: > > I am now running bsd.gdb with MP_LOCKDEBUG and WITNESS. > Here is an alternative version of the splassert traces from > during boot with line numbers. > > starting early daemons: syslogd pflogd unbound ntpd isakmpd. > starting RPC daemons:. > savecore: no core dump > checking quotas: done. > clearing /tmp > kern.securelevel: 0 -> 1 > creating runtime link editor directory cache. > preserving editor files. > starting network daemons: sshd snmpd dhcpd rad smtpd ftpproxy > ftpproxy6(failed) tftpproxy sndiod. > starting package daemons: lldpd dhcpcdsplassert: tdb_free: want 2 have 0 > Starting stack trace... > tdb_free(ffff800000f2db80) at tdb_free+0x47 [/sys/netinet/ip_ipsp.c:909] > pfkeyv2_send(fffffd805e21e3f8,ffff800000f27e00,170) at pfkeyv2_send+0x656 > [/sys/net/pfkeyv2.c:2141] > pfkeyv2_output(fffffd806731ea00,fffffd805e21e3f8,0,0) at pfkeyv2_output+0x8a > [/sys/net/pfkeyv2.c:442] > pfkeyv2_usrreq(fffffd805e21e3f8,9,fffffd806731ea00,0,0,ffff80001f6aefc8) at > pfkeyv2_usrreq+0x1b0 [/sys/net/pfkeyv2.c:394] > sosend(fffffd805e21e3f8,0,ffff80001f6fdd90,0,0,0) at sosend+0x3a9 > [/sys/kern/uipc_socket.c:597] > dofilewritev(ffff80001f6aefc8,7,ffff80001f6fdd90,0,ffff80001f6fde90) at > dofilewritev+0x14d [/sys/kern/sys_generic.c:382] > sys_writev(ffff80001f6aefc8,ffff80001f6fde30,ffff80001f6fde90) at > sys_writev+0xd2 [/sys/kern/sys_generic.c:328] > syscall(ffff80001f6fdf00) at syscall+0x3a9 [/sys/sys/syscall_mi.h:102] > Xsyscall() at Xsyscall+0x128 > end of kernel > end trace frame: 0x7f7ffffb31b0, count: 248 > End of stack trace. > splassert: tdb_free: want 2 have 0 > Starting stack trace... > tdb_free(ffff800000f2db80) at tdb_free+0x47 [/sys/netinet/ip_ipsp.c:909] > pfkeyv2_send(fffffd805e21e3f8,ffff800000f27e00,190) at pfkeyv2_send+0x656 > [/sys/net/pfkeyv2.c:2141] > pfkeyv2_output(fffffd8061acef00,fffffd805e21e3f8,0,0) at pfkeyv2_output+0x8a > [/sys/net/pfkeyv2.c:442] > pfkeyv2_usrreq(fffffd805e21e3f8,9,fffffd8061acef00,0,0,ffff80001f6aefc8) at > pfkeyv2_usrreq+0x1b0 [/sys/net/pfkeyv2.c:394] > sosend(fffffd805e21e3f8,0,ffff80001f6fdd90,0,0,0) at sosend+0x3a9 > [/sys/kern/uipc_socket.c:597] > dofilewritev(ffff80001f6aefc8,7,ffff80001f6fdd90,0,ffff80001f6fde90) at > dofilewritev+0x14d [/sys/kern/sys_generic.c:382] > sys_writev(ffff80001f6aefc8,ffff80001f6fde30,ffff80001f6fde90) at > sys_writev+0xd2 [/sys/kern/sys_generic.c:328] > syscall(ffff80001f6fdf00) at syscall+0x3a9 [/sys/sys/syscall_mi.h:102] > Xsyscall() at Xsyscall+0x128 > end of kernel > end trace frame: 0x7f7ffffb31b0, count: 248 > End of stack trace. > splassert: tdb_free: want 2 have 0 > Starting stack trace... > tdb_free(ffff800000f2db80) at tdb_free+0x47 [/sys/netinet/ip_ipsp.c:909] > pfkeyv2_send(fffffd805e21e3f8,ffff800000f27200,170) at pfkeyv2_send+0x656 > [/sys/net/pfkeyv2.c:2141] > pfkeyv2_output(fffffd8060d5f800,fffffd805e21e3f8,0,0) at pfkeyv2_output+0x8a > [/sys/net/pfkeyv2.c:442] > pfkeyv2_usrreq(fffffd805e21e3f8,9,fffffd8060d5f800,0,0,ffff80001f6aefc8) at > pfkeyv2_usrreq+0x1b0 [/sys/net/pfkeyv2.c:394] > sosend(fffffd805e21e3f8,0,ffff80001f6fdd90,0,0,0) at sosend+0x3a9 > [/sys/kern/uipc_socket.c:597] > dofilewritev(ffff80001f6aefc8,7,ffff80001f6fdd90,0,ffff80001f6fde90) at > dofilewritev+0x14d [/sys/kern/sys_generic.c:382] > sys_writev(ffff80001f6aefc8,ffff80001f6fde30,ffff80001f6fde90) at > sys_writev+0xd2 [/sys/kern/sys_generic.c:328] > syscall(ffff80001f6fdf00) at syscall+0x3a9 [/sys/sys/syscall_mi.h:102] > Xsyscall() at Xsyscall+0x128 > end of kernel > end trace frame: 0x7f7ffffb31b0, count: 248 > End of stack trace. > . > starting local daemons: cron. > Sat Nov 13 22:57:42 GMT 2021 > > OpenBSD/amd64 (jodrell.spacehopper.org) (tty00) > > login: reorder_kernel: failed -- see > /usr/share/relink/kernel/GENERIC.MP/relink.log > [-- sthen@localhost attached -- Sat Nov 13 22:55:14 2021] > > > isakmpd crashed; I flushed pfkey, restarted isakmpd and reloaded ipsec.conf, > then saw one more "splassert: tdb_free: want 2 have 0" after every new SA > > Now waiting for another kernel crash. >
^Msplassert: ipsp_ids_free: want 2 have 0 ^MStarting stack trace... ^Mipsp_ids_free(ffff800000f28e00) at ipsp_ids_free+0x4e [/sys/netinet/ip_ipsp.c:1129] ^Mtdb_free(ffff800000f2a888) at tdb_free+0x13c [/sys/netinet/ip_ipsp.c:929] ^Mpfkeyv2_send(fffffd8064a72d80,ffff800000f28b00,50) at pfkeyv2_send+0x656 [/sys/net/pfkeyv2.c:2141] ^Mpfkeyv2_output(fffffd8076187b00,fffffd8064a72d80,0,0) at pfkeyv2_output+0x8a [/sys/net/pfkeyv2.c:442] ^Mpfkeyv2_usrreq(fffffd8064a72d80,9,fffffd8076187b00,0,0,ffff80001f6afce8) at pfkeyv2_usrreq+0x1b0 [/sys/net/pfkeyv2.c:394] ^Msosend(fffffd8064a72d80,0,ffff80001f77fd30,0,0,0) at sosend+0x3a9 [/sys/kern/uipc_socket.c:597] ^Mdofilewritev(ffff80001f6afce8,7,ffff80001f77fd30,0,ffff80001f77fe30) at dofilewritev+0x14d [/sys/kern/sys_generic.c:382] ^Msys_writev(ffff80001f6afce8,ffff80001f77fdd0,ffff80001f77fe30) at sys_writev+0xd2 [/sys/kern/sys_generic.c:328] ^Msyscall(ffff80001f77fea0) at syscall+0x3a9 [/sys/sys/syscall_mi.h:102] ^MXsyscall() at Xsyscall+0x128 ^Mend of kernel ^Mend trace frame: 0x7f7ffffc3ff0, count: 247 ^MEnd of stack trace. ^Mpanic: kernel diagnostic assertion "refcnt != ~0" failed: file "/sys/kern/kern_ ^Msynch.c", line 824 ^MStopped at db_enter+0x10 [/sys/arch/amd64/amd64/db_interface.c:440]: p ^Mopq %rbp ^M TID PID UID PRFLAGS PFLAGS CPU COMMAND ^M 342644 33370 68 0x10 0x80 3 isakmpd ^M*358636 67323 0 0x14000 0x40000200 0K softclock ^Mdb_enter() at db_enter+0x10 [/sys/arch/amd64/amd64/db_interface.c:440] ^Mpanic(ffffffff81e54b7a) at panic+0xbf [/sys/kern/subr_prf.c:231] ^M__assert(ffffffff81ec035c,ffffffff81e274b8,338,ffffffff81e5902d) at __assert+0x ^M25 [/sys/kern/subr_prf.c:161] ^Mrefcnt_rele(ffff800000f2c670) at refcnt_rele+0x6f [/sys/kern/kern_synch.c:824] ^Mtdb_free(ffff800000f2c648) at tdb_free+0x116 [/sys/netinet/ip_ipsp.c:854] ^Mtdb_timeout(ffff800000f2c648) at tdb_timeout+0x39 [/sys/netinet/ip_ipsp.c:586] ^Mtimeout_run(ffff800000f2c6a0) at timeout_run+0x93 [/sys/kern/kern_timeout.c:207 ^M] ^Msoftclock_thread(ffff8000fffff500) at softclock_thread+0x11d [/sys/kern/kern_ti ^Mmeout.c:802] ^Mend trace frame: 0x0, count: 7 ^Mhttps://www.openbsd.org/ddb.html describes the minimum info required in bug ^Mreports. Insufficient info makes it difficult to find and fix bugs. ^Mddb{0}> ps ^M PID TID PPID UID S FLAGS WAIT COMMAND ^M 9687 13204 74983 1000 3 0x900483 kqread tail ^M 33370 342644 67266 68 7 0x90 isakmpd ^M 67266 330352 1 0 3 0x80 netio isakmpd ^M 74983 399516 21252 1000 3 0x10008b sigsusp ksh ^M 21252 425650 38609 1000 3 0x98 kqread sshd ^M 38609 372036 56630 0 3 0x92 kqread sshd ^M 20324 262588 1 0 3 0x100083 ttyin getty ^M 90119 277303 1 0 3 0x100098 kqread cron ^M 58265 363351 73338 846 3 0x100098 kqread dhcpcd ^M 2919 135701 73338 846 3 0x100098 kqread dhcpcd ^M 21023 23 73338 0 3 0x88 kqread dhcpcd ^M 73338 187898 1 846 3 0x100098 kqread dhcpcd ^M 38884 64762 11067 720 3 0x90 kqread lldpd ^M 11067 456543 1 0 3 0x80 netio lldpd ^M 12775 287323 1 99 3 0x100090 kqread sndiod ^M 13726 302476 1 110 3 0x100090 kqread sndiod ^M 23521 87877 15825 0 3 0x100090 kqread tftp-proxy ^M 15825 62223 1 108 3 0x90 kqread tftp-proxy ^M 85105 483454 1 109 3 0x90 kqread ftp-proxy ^M 86206 478101 65694 95 3 0x100092 kqread smtpd ^M 82807 291167 65694 103 3 0x100092 kqread smtpd ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^H 31521 344454 65694 95 3 0x100092 kqread smtpd ^M 20452 468317 65694 95 3 0x100092 kqread smtpd ^M 72867 194005 65694 95 3 0x100092 kqread smtpd ^M 22118 407997 65694 95 3 0x100092 kqread smtpd ^M 65694 350788 1 0 3 0x100080 kqread smtpd ^M 22486 136192 61819 94 3 0x100092 kqread rad ^M 58256 400280 61819 94 3 0x100092 kqread rad ^M 61819 460507 1 0 3 0x100080 kqread rad ^M 51364 474960 1 77 3 0x100090 kqread dhcpd ^M 63870 246442 1 0 3 0x100080 kqread snmpd ^M 34027 46645 1 91 3 0x92 kqread snmpd ^M 56630 456726 1 0 3 0x88 kqread sshd ^M 54370 255407 1 0 3 0x100080 kqread ntpd ^M 56913 484845 16720 83 3 0x100092 kqread ntpd ^M 16720 272340 1 83 3 0x100092 kqread ntpd ^M 12233 281136 97499 53 3 0x90 kqread unbound ^M 97499 271693 1 53 3 0x90 kqread unbound ^M 14891 206305 31956 74 3 0x100092 bpf pflogd ^M 31956 95382 1 0 3 0x80 netio pflogd ^M 56395 64162 52476 73 3 0x100090 kqread syslogd ^M 52476 280436 1 0 3 0x100082 netio syslogd ^M 54381 297690 1 0 3 0x100080 kqread resolvd ^M 67942 230839 37299 77 3 0x100092 kqread dhcpleased ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^H 16236 487836 37299 77 3 0x100092 kqread dhcpleased ^M 37299 151756 1 0 3 0x80 kqread dhcpleased ^M 70711 155058 0 0 3 0x14200 bored wg_crypt ^M 42386 502737 0 0 3 0x14200 bored wg_crypt ^M 2981 499833 0 0 3 0x14200 bored wg_crypt ^M 96546 135205 0 0 3 0x14200 bored wg_crypt ^M 83468 55493 0 0 3 0x14200 bored wg_handshake ^M 72007 252437 0 0 3 0x14200 bored wg_handshake ^M 45130 386772 62 115 3 0x100092 kqread slaacd ^M 95140 321693 62 115 3 0x100092 kqread slaacd ^M 62 191497 1 0 3 0x100080 kqread slaacd ^M 91596 460857 0 0 3 0x14200 bored smr ^M 85279 37518 0 0 3 0x14200 pgzero zerothread ^M 5652 32143 0 0 3 0x14200 aiodoned aiodoned ^M 93016 216478 0 0 3 0x14200 syncer update ^M 59629 140137 0 0 3 0x14200 cleaner cleaner ^M 53245 243160 0 0 3 0x14200 reaper reaper ^M 1526 307276 0 0 3 0x14200 pgdaemon pagedaemon ^M 4396 284304 0 0 3 0x14200 bored sensors ^M 17127 167462 0 0 3 0x14200 mmctsk sdmmc0 ^M 74873 115044 0 0 3 0x14200 usbtsk usbtask ^M 67518 406894 0 0 3 0x14200 usbatsk usbatsk ^M 65987 277497 0 0 3 0x40014200 acpi0 acpi0 ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^H 40494 73870 0 0 3 0x40014200 idle3 ^M 81457 368459 0 0 7 0x40014200 idle2 ^M 10648 120179 0 0 7 0x40014200 idle1 ^M 28631 519471 0 0 3 0x14200 bored softnet ^M 36813 217211 0 0 3 0x14200 bored systqmp ^M 24898 82912 0 0 3 0x14200 bored systq ^M*67323 358636 0 0 7 0x40014200 softclock ^M 59649 78911 0 0 3 0x40014200 idle0 ^M 1 505789 0 0 3 0x82 wait init ^M 0 0 -1 0 3 0x10200 scheduler swapper ^Mddb{0}> sh reg ^Mrdi 0 ^Mrsi 0x14 ^Mrbp 0xffff80001f5dbf60 ^Mrbx 0 ^Mrdx 0xde00000003ff0239 ^Mrcx 0x202 ^Mrax 0x62 ^Mr8 0x101010101010101 ^Mr9 0 ^Mr10 0 ^Mr11 0x35557c169bdb127 ^Mr12 0xffffffff82182a00 cpu_info_full_primary+0x2a00 ^Mr13 0xffff800000f2ca70 ^Mr14 0 ^Mr15 0xffffffff81e54b7a cmd0646_9_tim_udma+0x2acb8 ^Mrip 0xffffffff81138ad0 db_enter+0x10 ^Mcs 0x8 ^Mrflags 0x206 ^Mrsp 0xffff80001f5dbf60 ^Mss 0x10 ^Mdb_enter+0x10 [/sys/arch/amd64/amd64/db_interface.c:440]: popq %rbp ^Mddb{0}> sh all pools ^MName Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle ^Mplcache 128 44 0 0 2 0 2 2 0 8 0 ^Mrtpcb 120 26 0 17 1 0 1 1 0 8 0 ^Mrtentry 112 141 0 61 3 0 3 3 0 8 0 ^Munpcb 120 327 0 220 4 0 4 4 0 8 0 ^Msyncache 296 47 0 47 42 41 1 1 0 8 1 ^Mtcpqe 32 40 0 40 5 5 0 1 0 8 0 ^Mtcpcb 736 77 0 68 2 0 2 2 0 8 0 ^Mtdb 1088 27 0 20 2 1 1 1 0 8 0 ^Marp 120 84 0 61 1 0 1 1 0 8 0 ^Minpcb 304 13561 0 13494 45 39 6 12 0 8 0 ^Mnd6 48 8 0 0 1 0 1 1 0 8 0 ^Mipsec policy 336 12 0 6 2 1 1 1 0 8 0 ^Mpkpcb 40 5 0 4 2 1 1 1 0 8 0 ^Mcryptop 280 3838 0 3838 19 19 0 1 0 8 0 ^Mwgpeer 5448 5 0 0 1 0 1 1 0 8 0 ^Mwgaip 64 6 0 0 1 0 1 1 0 8 0 ^Mpfosfp 40 1428 0 1005 5 0 5 5 0 8 0 ^Mpfosfpen 112 1428 0 714 21 0 21 21 0 8 0 ^Mpfrke_plain 160 54 0 0 3 0 3 3 0 8 0 ^Mpfrktable 1344 12 0 6 1 0 1 1 0 8 0 ^Mpftag 88 1 0 0 1 0 1 1 0 8 0 ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^Hpfruleitem 16 53805 0 53047 22 13 9 19 0 8 0 ^Mpfstitem 24 53418 0 52572 39 26 13 32 0 8 0 ^Mpfstkey 112 53418 0 52571 307 248 59 151 0 8 0 ^Mpfstate 320 51385 0 50571 1232 1088 144 402 0 8 7 ^Mpfrule 1360 194 0 16 17 1 16 17 0 8 0 ^Mart_heap8 4096 4 0 0 4 0 4 4 0 8 0 ^Mart_heap4 256 306 0 4 19 0 19 19 0 8 0 ^Mart_table 32 310 0 4 3 0 3 3 0 8 0 ^Mart_node 16 141 0 62 1 0 1 1 0 8 0 ^Mdirhash 1024 93 0 40 9 2 7 7 0 8 0 ^Mdino2pl 256 3530 0 1678 131 15 116 116 0 8 0 ^Mffsino 272 3530 0 1678 140 16 124 124 0 8 0 ^Mnchpl 144 3887 0 1749 101 21 80 80 0 8 0 ^Mrtmask 32 11 0 4 1 0 1 1 0 8 0 ^Muvmvnodes 72 3555 0 0 65 0 65 65 0 8 0 ^Mvnodes 224 3555 0 0 210 0 210 210 0 8 0 ^Mnamei 1024 22955 0 22955 104 103 1 1 0 8 1 ^Mpercpumem 32 38 0 0 1 0 1 1 0 8 0 ^Maesni 576 18 0 11 1 0 1 1 1 8 0 ^Mehcixfer 296 40 0 38 1 0 1 1 0 8 0 ^Mxhcixfer 280 44 0 43 1 0 1 1 0 8 0 ^Mscxspl 216 22005 0 22005 38 37 1 2 0 8 1 ^Mplimitpl 152 54 0 24 2 0 2 2 0 8 0 ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^Hsigapl 424 983 0 907 10 0 10 10 0 8 0 ^Mfutexpl 56 3728 0 3728 1 1 0 1 0 8 0 ^Mknotepl 112 604 0 0 9 0 9 9 0 8 0 ^Mkqueuepl 216 145 0 94 4 1 3 4 0 8 0 ^Mpipepl 336 270 0 270 43 42 1 1 0 8 1 ^Mfdescpl 496 958 0 907 8 0 8 8 0 8 0 ^Mfilepl 152 21997 0 21729 23 12 11 13 0 8 0 ^Mlockfpl 104 42 0 38 1 0 1 1 0 8 0 ^Mlockfspl 48 23 0 19 1 0 1 1 0 8 0 ^Msessionpl 144 74 0 47 2 0 2 2 0 8 0 ^Mpgrppl 48 109 0 81 1 0 1 1 0 8 0 ^Mucredpl 96 404 0 352 2 0 2 2 0 8 0 ^Mzombiepl 144 907 0 907 42 41 1 1 0 8 1 ^Mprocesspl 1072 983 0 907 6 0 6 6 0 8 0 ^Mprocpl 672 995 0 919 7 0 7 7 0 8 0 ^Msockpl 480 13919 0 13735 95 71 24 33 0 8 0 ^Mmcl8k 8192 3 0 0 1 0 1 1 0 8 0 ^Mmcl4k 4096 20 0 0 3 0 3 3 0 8 0 ^Mmcl2k2 2112 480 0 0 12 1 11 12 0 8 0 ^Mmcl2k 2048 205 0 0 10 3 7 10 0 8 0 ^Mmtagpl 96 478 0 0 3 1 2 3 0 8 0 ^Mmbufpl 256 1000 0 0 27 1 26 27 0 8 0 ^Mbufpl 280 19443 0 4192 1102 12 1090 1090 0 8 0 ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^Hanonpl 24 159407 0 137534 171 37 134 137 0 174 1 ^Mamapchunkpl 152 14047 0 12639 74 19 55 56 0 158 0 ^Mamappl16 200 461 0 430 2 0 2 2 0 8 0 ^Mamappl15 192 51 0 50 1 0 1 1 0 8 0 ^Mamappl14 184 8 0 8 1 1 0 1 0 8 0 ^Mamappl13 176 61 0 60 2 1 1 1 0 8 0 ^Mamappl12 168 8 0 8 1 1 0 1 0 8 0 ^Mamappl11 160 177 0 111 3 0 3 3 0 8 0 ^Mamappl10 152 72 0 70 1 0 1 1 0 8 0 ^Mamappl9 144 28 0 26 1 0 1 1 0 8 0 ^Mamappl8 136 1149 0 1105 2 0 2 2 0 8 0 ^Mamappl7 128 322 0 265 3 1 2 3 0 8 0 ^Mamappl6 120 442 0 412 2 1 1 2 0 8 0 ^Mamappl5 112 570 0 457 4 0 4 4 0 8 0 ^Mamappl4 104 2573 0 2216 10 0 10 10 0 8 0 ^Mamappl3 96 498 0 477 4 3 1 4 0 8 0 ^Mamappl2 88 3358 0 2988 20 11 9 13 0 8 0 ^Mamappl1 80 34008 0 32305 110 72 38 42 0 8 2 ^Mamappl 88 6200 0 5717 12 1 11 12 0 92 0 ^Mdma4096 4096 1 0 1 1 1 0 1 0 8 0 ^Mdma512 512 3 0 2 1 0 1 1 0 8 0 ^Mdma256 256 10 0 10 1 1 0 1 0 8 0 ^Mdma128 128 1 0 1 1 1 0 1 0 8 0 ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^Hdma64 64 10 0 10 1 1 0 1 0 8 0 ^Mdma32 32 9 0 9 1 1 0 1 0 8 0 ^Mdma16 16 29 0 29 1 1 0 1 0 8 0 ^Maobjpl 64 2 0 0 1 0 1 1 0 8 0 ^Muaddrrnd 24 958 0 907 1 0 1 1 0 8 0 ^Muaddrbest 32 2 0 0 1 0 1 1 0 8 0 ^Muaddr 24 958 0 907 1 0 1 1 0 8 0 ^Mvmmpekpl 168 57451 0 57433 2 1 1 2 0 8 0 ^Mvmmpepl 168 104716 0 99897 606 384 222 243 0 357 10 ^Mvmsppl 368 957 0 907 6 0 6 6 0 8 0 ^Mrwobjpl 56 33636 0 30668 130 86 44 47 0 8 2 ^Mpdppl 4096 957 0 907 152 100 52 56 0 8 2 ^Mpvpl 32 641571 0 607155 644 354 290 309 0 265 8 ^Mpmappl 224 957 0 907 4 0 4 4 0 8 0 ^Mextentpl 40 72 0 45 1 0 1 1 0 8 0 ^Mphpool 112 675 0 287 12 0 12 12 0 8 0 ^Mddb{0}> show panic ^M*cpu0: kernel diagnostic assertion "refcnt != ~0" failed: file "/sys/kern/kern_ ^Msynch.c", line 824 ^Mddb{0}> mach ddbcpu 1 ^MStopped at x86_ipi_db+0x12 [/sys/arch/amd64/amd64/db_interface.c:393]: l ^Meave ^Mx86_ipi_db(ffff80001f448ff0) at x86_ipi_db+0x12 [/sys/arch/amd64/amd64/db_inter ^Mface.c:393] ^Mx86_ipi_handler() at x86_ipi_handler+0x80 [/sys/arch/amd64/amd64/ipi.c:106] ^MXresume_lapic_ipi() at Xresume_lapic_ipi+0x23 ^Macpicpu_idle() at acpicpu_idle+0x11f [/sys/dev/acpi/acpicpu.c:1270] ^Msched_idle(ffff80001f448ff0) at sched_idle+0x27e [/sys/kern/kern_sched.c:178] ^Mend trace frame: 0x0, count: 10 ^Mddb{1}> ps /o ^M TID PID UID PRFLAGS PFLAGS CPU COMMAND ^M 342644 33370 68 0x10 0x80 3 isakmpd ^M 358636 67323 0 0x14000 0x40000200 0K softclock ^Mddb{1}> mach ddbcpu 2 ^MStopped at x86_ipi_db+0x12 [/sys/arch/amd64/amd64/db_interface.c:393]: l ^Meave ^Mx86_ipi_db(ffff80001f451ff0) at x86_ipi_db+0x12 [/sys/arch/amd64/amd64/db_inter ^Mface.c:393] ^Mx86_ipi_handler() at x86_ipi_handler+0x80 [/sys/arch/amd64/amd64/ipi.c:106] ^MXresume_lapic_ipi() at Xresume_lapic_ipi+0x23 ^Macpicpu_idle() at acpicpu_idle+0x11f [/sys/dev/acpi/acpicpu.c:1270] ^Msched_idle(ffff80001f451ff0) at sched_idle+0x27e [/sys/kern/kern_sched.c:178] ^Mend trace frame: 0x0, count: 10 ^Mddb{2}> mach ddbcpu 3 ^MStopped at x86_ipi_db+0x12 [/sys/arch/amd64/amd64/db_interface.c:393]: l ^Meave ^Mx86_ipi_db(ffff80001f45aff0) at x86_ipi_db+0x12 [/sys/arch/amd64/amd64/db_inter ^Mface.c:393] ^Mx86_ipi_handler() at x86_ipi_handler+0x80 [/sys/arch/amd64/amd64/ipi.c:106] ^MXresume_lapic_ipi() at Xresume_lapic_ipi+0x23 ^M__mp_lock(ffffffff82335a58) at __mp_lock+0xbe [/sys/kern/kern_lock.c:120] ^M__mp_acquire_count(ffffffff82335a58,3) at __mp_acquire_count+0x38 [/sys/kern/ke ^Mrn_lock.c:227] ^Mmi_switch() at mi_switch+0x299 [/sys/kern/sched_bsd.c:417] ^Msleep_finish(ffff80001f77f860,1) at sleep_finish+0x11c [/sys/kern/kern_synch.c: ^M427] ^Mmsleep(fffffd805d1d86d0,fffffd805d1d86d0,318,ffffffff81eb5e2b,1d2ab) at msleep+ ^M0xcc [/sys/kern/kern_synch.c:259] ^Mkqueue_sleep(fffffd805d1d86d0,ffff80001f77fd60) at kqueue_sleep+0xbe [/sys/kern ^M/kern_event.c:1291] ^Mkqueue_scan(ffff80001f77fb10,8,ffff80001f77fa10,ffff80001f77fd60,ffff80001f6afc ^Me8,ffff80001f77fccc) at kqueue_scan+0xfc [/sys/kern/kern_event.c:1349] ^Mdopselect(ffff80001f6afce8,23,b3898868b50,b390aa026a0,0,ffff80001f77fd60,ea70eb ^Mf1baf9bb17,ffff80001f6afce8) at dopselect+0x388 [/sys/kern/sys_generic.c:691] ^Msys_pselect(ffff80001f6afce8,ffff80001f77fdd0,ffff80001f77fe30) at sys_pselect+ ^M0xdb [/sys/kern/sys_generic.c:595] ^M--db_more--^H^H^H^H^H^H^H^H^H^H^H ^H^H^H^H^H^H^H^H^H^H^Hsyscall(ffff80001f77fea0) at syscall+0x3a9 [/sys/sys/syscall_mi.h:102] ^MXsyscall() at Xsyscall+0x128 ^Mend of kernel ^Mend trace frame: 0x7f7ffffc41a0, count: 1 ^Mddb{3}> t ^H ^Hr /t 0t342644 ^Msleep_finish(ffff80001f77f860,1) at sleep_finish+0x11c [/sys/kern/kern_synch.c: ^M427] ^Mmsleep(fffffd805d1d86d0,fffffd805d1d86d0,318,ffffffff81eb5e2b,1d2ab) at msleep+ ^M0xcc [/sys/kern/kern_synch.c:259] ^Mkqueue_sleep(fffffd805d1d86d0,ffff80001f77fd60) at kqueue_sleep+0xbe [/sys/kern ^M/kern_event.c:1291] ^Mkqueue_scan(ffff80001f77fb10,8,ffff80001f77fa10,ffff80001f77fd60,ffff80001f6afc ^Me8,ffff80001f77fccc) at kqueue_scan+0xfc [/sys/kern/kern_event.c:1349] ^Mdopselect(ffff80001f6afce8,23,b3898868b50,b390aa026a0,0,ffff80001f77fd60,ea70eb ^Mf1baf9bb17,ffff80001f6afce8) at dopselect+0x388 [/sys/kern/sys_generic.c:691] ^Msys_pselect(ffff80001f6afce8,ffff80001f77fdd0,ffff80001f77fe30) at sys_pselect+ ^M0xdb [/sys/kern/sys_generic.c:595] ^Msyscall(ffff80001f77fea0) at syscall+0x3a9 [/sys/sys/syscall_mi.h:102] ^MXsyscall() at Xsyscall+0x128 ^Mend of kernel ^Mend trace frame: 0x7f7ffffc41a0, count: -8 ^Mddb{3}> ^H ^Hbo t^H ^Hr ^Mrebooting... ^MPC Engines apu2^M coreboot build 20202905^M BIOS version v4.12.0.1^M 2032 MB DRAM^M
