Found it in dmesg buffer:
Stopped at Debugger+0x9: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb{0}> trace
Debugger() at Debugger+0x9
panic() at panic+0xfe
mtx_enter() at mtx_enter+0x60
sofree() at sofree+0xa0
in_pcbdetach() at in_pcbdetach+0x40
tcp_close() at tcp_close+0xad
tcp_timer_2msl() at tcp_timer_2msl+0x90
softclock() at softclock+0x315
softintr_dispatch() at softintr_dispatch+0x8b
Xsoftclock() at Xsoftclock+0x1f
--- interrupt ---
(null)() at 0x8
end of kernel
end trace frame: 0x1102000000000001, count: -11
ddb{0}> show registers
rdi 0x1
rsi 0x282
rbp 0xffff8000221c68f8
rbx 0xffffffff813285e0 mtx_enter+0x60
rdx 0
rcx 0xffffffff8188c640 cpu_info_primary
rax 0x1
r8 0xffff8000221c6818
r9 0x1
r10 0
r11 0xffff8000221c66b0
r12 0x100
r13 0xffff8000221c6908
r14 0xffffff011d088010
r15 0
rip 0xffffffff81343b09 Debugger+0x9
cs 0x8
rflags 0x286
rsp 0xffff8000221c68e8
ss 0x10
Debugger+0x9: leave
ddb{0}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
19921 12801 19921 0 3 0x83 poll systat
12801 569 12801 0 3 0x8b pause ksh
569 23137 569 0 3 0x92 select sshd
13678 1 13678 0 3 0x83 ttyin getty
9776 1 9776 0 3 0x83 ttyin getty
3392 1 3392 0 3 0x83 ttyin getty
24230 1 24230 0 3 0x83 ttyin getty
14469 1 14469 0 3 0x83 ttyin getty
29209 1 29209 0 3 0x80 poll cron
4796 4087 4087 95 3 0x90 kqread smtpd
269 4087 4087 95 3 0x90 kqread smtpd
28144 4087 4087 95 3 0x90 kqread smtpd
13626 4087 4087 95 3 0x90 kqread smtpd
4756 4087 4087 95 3 0x90 kqread smtpd
23276 4087 4087 103 3 0x90 kqread smtpd
4087 1 4087 0 3 0x80 kqread smtpd
29277 28344 28344 89 3 0x90 kqread relayd
15361 28344 28344 89 3 0x90 kqread relayd
28344 26987 28344 89 3 0x90 kqread relayd
*18801 1528 1528 89 7 0x10 relayd
15066 1528 1528 89 3 0x90 kqread relayd
1528 26987 1528 89 3 0x90 kqread relayd
14013 26987 14013 89 3 0x90 kqread relayd
25397 26987 25397 89 3 0x90 kqread relayd
26987 1 26987 0 3 0x80 kqread relayd
23945 11110 11110 85 3 0x90 kqread ospfd
12948 11110 11110 85 3 0x90 kqread ospfd
11110 1 11110 0 3 0x80 kqread ospfd
23137 1 23137 0 3 0x80 select sshd
10031 27507 981 83 3 0x90 poll ntpd
27507 981 981 83 3 0x90 poll ntpd
981 1 981 0 3 0x80 poll ntpd
12220 25415 25415 74 3 0x90 bpf pflogd
25415 1 25415 0 3 0x80 netio pflogd
3275 32486 32486 73 3 0x90 kqread syslogd
32486 1 32486 0 3 0x80 netio syslogd
14861 0 0 0 3 0x14200 pgzero zerothread
24670 0 0 0 3 0x14200 aiodoned aiodoned
29165 0 0 0 3 0x14200 syncer update
27875 0 0 0 3 0x14200 cleaner cleaner
645 0 0 0 3 0x14200 reaper reaper
17692 0 0 0 3 0x14200 pgdaemon pagedaemon
766 0 0 0 3 0x14200 bored crypto
4241 0 0 0 3 0x14200 pftm pfpurge
9257 0 0 0 3 0x40014200 acpi0 acpi0
25569 0 0 0 7 0x40014200 idle1
7980 0 0 0 3 0x14200 bored sensors
3321 0 0 0 3 0x14200 bored softnet
4749 0 0 0 3 0x14200 bored systqmp
16058 0 0 0 3 0x14200 bored systq
15954 0 0 0 3 0x40014200 idle0
1 0 1 0 3 0x82 wait init
0 -1 0 0 3 0x10200 scheduler swapper
> On 4 feb. 2016, at 13:49, mxb <m...@alumni.chalmers.se> wrote:
>
>
> I was able to re-produce this panic with similar stack trace.
> Unfortunately 'trace/show regs/ps' are not in txt format, but are
screenshots.
>
> //mxb
>
>> On 4 feb. 2016, at 12:42, mxb <m...@alumni.chalmers.se> wrote:
>>
>>
>> Hey,
>> see those again on 5.8-STABLE.
>>
>> This is a 2-node CARP setup within VMWare ESX.
>> Both machines are rebooting after this and it happens quite often.
>>
>> Any ideas?
>>
>> panic: mtx_enter: locking against myself
>> Starting stack trace...
>> panic() at panic+0x10b
>> mtx_enter() at mtx_enter+0x60
>> sofree() at sofree+0xa0
>> in_pcbdetach() at in_pcbdetach+0x40
>> tcp_close() at tcp_close+0xad
>> tcp_timer_2msl() at tcp_timer_2msl+0x90
>> softclock() at softclock+0x315
>> softintr_dispatch() at softintr_dispatch+0x8b
>> Xsoftclock() at Xsoftclock+0x1f
>> --- interrupt ---
>> (null)() at 0x8
>> (null)() at 0xffffff0118c50f90
>> end trace frame: 0x0, count: 246
>> End of stack trace.
>> syncing disks... panic: assertwaitok: non-zero mutex count: 1
>> Starting stack trace...
>> panic() at panic+0x10b
>> assertwaitok() at assertwaitok+0x52
>> bufq_wait() at bufq_wait+0x2d
>> bwrite() at bwrite+0xfe
>> VOP_BWRITE() at VOP_BWRITE+0x38
>> ffs_fsync() at ffs_fsync+0x13f
>> VOP_FSYNC() at VOP_FSYNC+0x3c
>> ffs_sync() at ffs_sync+0xc4
>> sys_sync() at sys_sync+0x87
>> vfs_syncwait() at vfs_syncwait+0x50
>> vfs_shutdown() at panic: mtx_enter: locking against myself
>> Faulted in traceback, aborting…
>>
>> OpenBSD 5.8-stable (GENERIC.MP) #0: Thu Oct 22 18:52:06 CEST 2015
>> root@prdlba0001:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>> real mem = 4278124544 (4079MB)
>> avail mem = 4144574464 (3952MB)
>> mpath0 at root
>> scsibus0 at mpath0: 256 targets
>> mainbus0 at root
>> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (364 entries)
>> bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014
>> bios0: VMware, Inc. VMware Virtual Platform
>> acpi0 at bios0: rev 2
>> acpi0: sleep states S0 S1 S4 S5
>> acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
>> acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S3F0(S3)
S4F0(S3) S5F0(S3) S6F0(S3) S7F0(S3) S8F0(S3) S9F0(S3) S10F(S3) S11F(S3)
S12F(S3) S13F(S3) [...]
>> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>> cpu0 at mainbus0: apid 0 (boot processor)
>> cpu0: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2400.20 MHz
>> cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,PERF,I
TSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
>> cpu0: 256KB 64b/line 8-way L2 cache
>> cpu0: smt 0, core 0, package 0
>> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
>> cpu0: apic clock running at 65MHz
>> cpu1 at mainbus0: apid 1 (application processor)
>> cpu1: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2399.74 MHz
>> cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,PERF,I
TSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
>> cpu1: 256KB 64b/line 8-way L2 cache
>> cpu1: smt 0, core 1, package 0
>> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
>> acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127
>> acpihpet0 at acpi0: 14318179 Hz
>> acpiprt0 at acpi0: bus 0 (PCI0)
>> acpicpu0 at acpi0: C1(@1 halt!)
>> acpicpu1 at acpi0: C1(@1 halt!)
>> acpibat0 at acpi0: BAT1 not present
>> acpibat1 at acpi0: BAT2 not present
>> acpiac0 at acpi0: AC unit online
>> acpibtn0 at acpi0: SLPB
>> acpibtn1 at acpi0: LID_
>> pvbus0 at mainbus0: VMware
>> vmt0 at pvbus0
>> pci0 at mainbus0 bus 0
>> pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
>> ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
>> pci1 at ppb0 bus 1
>> pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
>> pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel
0 configured to compatibility, channel 1 configured to compatibility
>> pciide0: channel 0 disabled (no drives)
>> pciide0: channel 1 disabled (no drives)
>> piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus
disabled
>> "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
>> vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
>> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
>> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
>> ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
>> pci2 at ppb1 bus 2
>> ahci0 at pci2 dev 1 function 0 "VMware AHCI" rev 0x00: apic 2 int 19, AHCI
1.3
>> ahci0: port 0: 6.0Gb/s
>> scsibus1 at ahci0: 32 targets
>> cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware SATA CD00, 1.00> ATAPI
5/cdrom removable
>> ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
>> pci3 at ppb2 bus 3
>> vmwpvs0 at pci3 dev 0 function 0 "VMware PVSCSI" rev 0x02: apic 2 int 18
>> scsibus2 at vmwpvs0: 16 targets
>> sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct
fixed
>> sd0: 16384MB, 512 bytes/sector, 33554432 sectors
>> ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
>> pci4 at ppb3 bus 4
>> ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
>> pci5 at ppb4 bus 5
>> ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
>> pci6 at ppb5 bus 6
>> ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
>> pci7 at ppb6 bus 7
>> ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
>> pci8 at ppb7 bus 8
>> ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
>> pci9 at ppb8 bus 9
>> ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
>> pci10 at ppb9 bus 10
>> ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
>> pci11 at ppb10 bus 11
>> vmx0 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:a9:8f
>> ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
>> pci12 at ppb11 bus 12
>> vmx1 at pci12 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:3a:8a
>> ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
>> pci13 at ppb12 bus 13
>> vmx2 at pci13 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:6d:7d
>> ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
>> pci14 at ppb13 bus 14
>> vmx3 at pci14 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:b5:57
>> ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
>> pci15 at ppb14 bus 15
>> ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
>> pci16 at ppb15 bus 16
>> ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
>> pci17 at ppb16 bus 17
>> ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
>> pci18 at ppb17 bus 18
>> ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
>> pci19 at ppb18 bus 19
>> vmx4 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:62:2e
>> ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
>> pci20 at ppb19 bus 20
>> vmx5 at pci20 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:e5:4e
>> ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
>> pci21 at ppb20 bus 21
>> vmx6 at pci21 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:b3:41
>> ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
>> pci22 at ppb21 bus 22
>> vmx7 at pci22 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:98:e0
>> ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
>> pci23 at ppb22 bus 23
>> ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
>> pci24 at ppb23 bus 24
>> ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
>> pci25 at ppb24 bus 25
>> ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
>> pci26 at ppb25 bus 26
>> ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
>> pci27 at ppb26 bus 27
>> vmx8 at pci27 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 17,
address 00:50:56:92:04:05
>> ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
>> pci28 at ppb27 bus 28
>> vmx9 at pci28 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 17,
address 00:50:56:92:2c:ed
>> ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
>> pci29 at ppb28 bus 29
>> ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
>> pci30 at ppb29 bus 30
>> ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
>> pci31 at ppb30 bus 31
>> ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
>> pci32 at ppb31 bus 32
>> ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
>> pci33 at ppb32 bus 33
>> ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
>> pci34 at ppb33 bus 34
>> isa0 at pcib0
>> isadma0 at isa0
>> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>> pckbd0 at pckbc0 (kbd slot)
>> wskbd0 at pckbd0: console keyboard, using wsdisplay0
>> pms0 at pckbc0 (aux slot)
>> wsmouse0 at pms0 mux 0
>> pcppi0 at isa0 port 0x61
>> spkr0 at pcppi0
>> lpt0 at isa0 port 0x378/4 irq 7
>> vscsi0 at root
>> scsibus3 at vscsi0: 256 targets
>> softraid0 at root
>> scsibus4 at softraid0: 256 targets
>> root on sd0a (30d13733f6764d22.a) swap on sd0b dump on sd0b
>> WARNING: / was not properly unmounted
>> carp: pfsync0 demoted group carp by 32 to 160 (pfsync init)
>> carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init)
>> carp: pfsync0 demoted group carp by 1 to 161 (pfsync bulk start)
>> carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start)
>> carp239: state transition: BACKUP -> MASTER
>> carp241: state transition: BACKUP -> MASTER
>> carp243: state transition: BACKUP -> MASTER
>> carp: pfsync0 demoted group carp by -1 to 32 (pfsync bulk done)
>> carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk done)
>> carp: pfsync0 demoted group carp by -32 to 0 (pfsync init)
>> carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)
>> carp238: state transition: BACKUP -> MASTER
>> carp240: state transition: BACKUP -> MASTER
>> carp242: state transition: BACKUP -> MASTER
>> carp244: state transition: BACKUP -> MASTER
>> carp302: state transition: BACKUP -> MASTER
>> carp238: state transition: MASTER -> BACKUP
>> carp240: state transition: MASTER -> BACKUP
>> carp242: state transition: MASTER -> BACKUP
>> carp302: state transition: MASTER -> BACKUP
>> carp244: state transition: MASTER -> BACKUP
