Any one from @devs have time to pick it up?
This is a new env. , so I have time to investigate.
Access can be provided on need bases.
//mxb
> On 4 feb. 2016, at 15:46, mxb <m...@alumni.chalmers.se> wrote:
>
> Found it in dmesg buffer:
>
> Stopped at Debugger+0x9: leave
> RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
> IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO.
> DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
>
> ddb{0}> trace
> Debugger() at Debugger+0x9
> panic() at panic+0xfe
> mtx_enter() at mtx_enter+0x60
> sofree() at sofree+0xa0
> in_pcbdetach() at in_pcbdetach+0x40
> tcp_close() at tcp_close+0xad
> tcp_timer_2msl() at tcp_timer_2msl+0x90
> softclock() at softclock+0x315
> softintr_dispatch() at softintr_dispatch+0x8b
> Xsoftclock() at Xsoftclock+0x1f
> --- interrupt ---
> (null)() at 0x8
> end of kernel
> end trace frame: 0x1102000000000001, count: -11
>
> ddb{0}> show registers
> rdi 0x1
> rsi 0x282
> rbp 0xffff8000221c68f8
> rbx 0xffffffff813285e0 mtx_enter+0x60
> rdx 0
> rcx 0xffffffff8188c640 cpu_info_primary
> rax 0x1
> r8 0xffff8000221c6818
> r9 0x1
> r10 0
> r11 0xffff8000221c66b0
> r12 0x100
> r13 0xffff8000221c6908
> r14 0xffffff011d088010
> r15 0
> rip 0xffffffff81343b09 Debugger+0x9
> cs 0x8
> rflags 0x286
> rsp 0xffff8000221c68e8
> ss 0x10
> Debugger+0x9: leave
>
> ddb{0}> ps
> PID PPID PGRP UID S FLAGS WAIT COMMAND
> 19921 12801 19921 0 3 0x83 poll systat
> 12801 569 12801 0 3 0x8b pause ksh
> 569 23137 569 0 3 0x92 select sshd
> 13678 1 13678 0 3 0x83 ttyin getty
> 9776 1 9776 0 3 0x83 ttyin getty
> 3392 1 3392 0 3 0x83 ttyin getty
> 24230 1 24230 0 3 0x83 ttyin getty
> 14469 1 14469 0 3 0x83 ttyin getty
> 29209 1 29209 0 3 0x80 poll cron
> 4796 4087 4087 95 3 0x90 kqread smtpd
> 269 4087 4087 95 3 0x90 kqread smtpd
> 28144 4087 4087 95 3 0x90 kqread smtpd
> 13626 4087 4087 95 3 0x90 kqread smtpd
> 4756 4087 4087 95 3 0x90 kqread smtpd
> 23276 4087 4087 103 3 0x90 kqread smtpd
> 4087 1 4087 0 3 0x80 kqread smtpd
> 29277 28344 28344 89 3 0x90 kqread relayd
> 15361 28344 28344 89 3 0x90 kqread relayd
> 28344 26987 28344 89 3 0x90 kqread relayd
> *18801 1528 1528 89 7 0x10 relayd
> 15066 1528 1528 89 3 0x90 kqread relayd
> 1528 26987 1528 89 3 0x90 kqread relayd
> 14013 26987 14013 89 3 0x90 kqread relayd
> 25397 26987 25397 89 3 0x90 kqread relayd
> 26987 1 26987 0 3 0x80 kqread relayd
> 23945 11110 11110 85 3 0x90 kqread ospfd
> 12948 11110 11110 85 3 0x90 kqread ospfd
> 11110 1 11110 0 3 0x80 kqread ospfd
> 23137 1 23137 0 3 0x80 select sshd
> 10031 27507 981 83 3 0x90 poll ntpd
> 27507 981 981 83 3 0x90 poll ntpd
> 981 1 981 0 3 0x80 poll ntpd
> 12220 25415 25415 74 3 0x90 bpf pflogd
> 25415 1 25415 0 3 0x80 netio pflogd
> 3275 32486 32486 73 3 0x90 kqread syslogd
> 32486 1 32486 0 3 0x80 netio syslogd
> 14861 0 0 0 3 0x14200 pgzero zerothread
> 24670 0 0 0 3 0x14200 aiodoned aiodoned
> 29165 0 0 0 3 0x14200 syncer update
> 27875 0 0 0 3 0x14200 cleaner cleaner
> 645 0 0 0 3 0x14200 reaper reaper
> 17692 0 0 0 3 0x14200 pgdaemon pagedaemon
> 766 0 0 0 3 0x14200 bored crypto
> 4241 0 0 0 3 0x14200 pftm pfpurge
> 9257 0 0 0 3 0x40014200 acpi0 acpi0
> 25569 0 0 0 7 0x40014200 idle1
> 7980 0 0 0 3 0x14200 bored sensors
> 3321 0 0 0 3 0x14200 bored softnet
> 4749 0 0 0 3 0x14200 bored systqmp
> 16058 0 0 0 3 0x14200 bored systq
> 15954 0 0 0 3 0x40014200 idle0
> 1 0 1 0 3 0x82 wait init
> 0 -1 0 0 3 0x10200 scheduler swapper
>
>> On 4 feb. 2016, at 13:49, mxb <m...@alumni.chalmers.se> wrote:
>>
>>
>> I was able to re-produce this panic with similar stack trace.
>> Unfortunately 'trace/show regs/ps' are not in txt format, but are
screenshots.
>>
>> //mxb
>>
>>> On 4 feb. 2016, at 12:42, mxb <m...@alumni.chalmers.se> wrote:
>>>
>>>
>>> Hey,
>>> see those again on 5.8-STABLE.
>>>
>>> This is a 2-node CARP setup within VMWare ESX.
>>> Both machines are rebooting after this and it happens quite often.
>>>
>>> Any ideas?
>>>
>>> panic: mtx_enter: locking against myself
>>> Starting stack trace...
>>> panic() at panic+0x10b
>>> mtx_enter() at mtx_enter+0x60
>>> sofree() at sofree+0xa0
>>> in_pcbdetach() at in_pcbdetach+0x40
>>> tcp_close() at tcp_close+0xad
>>> tcp_timer_2msl() at tcp_timer_2msl+0x90
>>> softclock() at softclock+0x315
>>> softintr_dispatch() at softintr_dispatch+0x8b
>>> Xsoftclock() at Xsoftclock+0x1f
>>> --- interrupt ---
>>> (null)() at 0x8
>>> (null)() at 0xffffff0118c50f90
>>> end trace frame: 0x0, count: 246
>>> End of stack trace.
>>> syncing disks... panic: assertwaitok: non-zero mutex count: 1
>>> Starting stack trace...
>>> panic() at panic+0x10b
>>> assertwaitok() at assertwaitok+0x52
>>> bufq_wait() at bufq_wait+0x2d
>>> bwrite() at bwrite+0xfe
>>> VOP_BWRITE() at VOP_BWRITE+0x38
>>> ffs_fsync() at ffs_fsync+0x13f
>>> VOP_FSYNC() at VOP_FSYNC+0x3c
>>> ffs_sync() at ffs_sync+0xc4
>>> sys_sync() at sys_sync+0x87
>>> vfs_syncwait() at vfs_syncwait+0x50
>>> vfs_shutdown() at panic: mtx_enter: locking against myself
>>> Faulted in traceback, aborting…
>>>
>>> OpenBSD 5.8-stable (GENERIC.MP) #0: Thu Oct 22 18:52:06 CEST 2015
>>> root@prdlba0001:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>> real mem = 4278124544 (4079MB)
>>> avail mem = 4144574464 (3952MB)
>>> mpath0 at root
>>> scsibus0 at mpath0: 256 targets
>>> mainbus0 at root
>>> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (364 entries)
>>> bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014
>>> bios0: VMware, Inc. VMware Virtual Platform
>>> acpi0 at bios0: rev 2
>>> acpi0: sleep states S0 S1 S4 S5
>>> acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
>>> acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3)
S3F0(S3) S4F0(S3) S5F0(S3) S6F0(S3) S7F0(S3) S8F0(S3) S9F0(S3) S10F(S3)
S11F(S3) S12F(S3) S13F(S3) [...]
>>> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>>> cpu0 at mainbus0: apid 0 (boot processor)
>>> cpu0: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2400.20 MHz
>>> cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,PERF,I
TSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
>>> cpu0: 256KB 64b/line 8-way L2 cache
>>> cpu0: smt 0, core 0, package 0
>>> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
>>> cpu0: apic clock running at 65MHz
>>> cpu1 at mainbus0: apid 1 (application processor)
>>> cpu1: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2399.74 MHz
>>> cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,PERF,I
TSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
>>> cpu1: 256KB 64b/line 8-way L2 cache
>>> cpu1: smt 0, core 1, package 0
>>> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
>>> acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127
>>> acpihpet0 at acpi0: 14318179 Hz
>>> acpiprt0 at acpi0: bus 0 (PCI0)
>>> acpicpu0 at acpi0: C1(@1 halt!)
>>> acpicpu1 at acpi0: C1(@1 halt!)
>>> acpibat0 at acpi0: BAT1 not present
>>> acpibat1 at acpi0: BAT2 not present
>>> acpiac0 at acpi0: AC unit online
>>> acpibtn0 at acpi0: SLPB
>>> acpibtn1 at acpi0: LID_
>>> pvbus0 at mainbus0: VMware
>>> vmt0 at pvbus0
>>> pci0 at mainbus0 bus 0
>>> pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
>>> ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
>>> pci1 at ppb0 bus 1
>>> pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
>>> pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
>>> pciide0: channel 0 disabled (no drives)
>>> pciide0: channel 1 disabled (no drives)
>>> piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus
disabled
>>> "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
>>> vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
>>> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
>>> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
>>> ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
>>> pci2 at ppb1 bus 2
>>> ahci0 at pci2 dev 1 function 0 "VMware AHCI" rev 0x00: apic 2 int 19, AHCI
1.3
>>> ahci0: port 0: 6.0Gb/s
>>> scsibus1 at ahci0: 32 targets
>>> cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware SATA CD00, 1.00> ATAPI
5/cdrom removable
>>> ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
>>> pci3 at ppb2 bus 3
>>> vmwpvs0 at pci3 dev 0 function 0 "VMware PVSCSI" rev 0x02: apic 2 int 18
>>> scsibus2 at vmwpvs0: 16 targets
>>> sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct
fixed
>>> sd0: 16384MB, 512 bytes/sector, 33554432 sectors
>>> ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
>>> pci4 at ppb3 bus 4
>>> ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
>>> pci5 at ppb4 bus 5
>>> ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
>>> pci6 at ppb5 bus 6
>>> ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
>>> pci7 at ppb6 bus 7
>>> ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
>>> pci8 at ppb7 bus 8
>>> ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
>>> pci9 at ppb8 bus 9
>>> ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
>>> pci10 at ppb9 bus 10
>>> ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
>>> pci11 at ppb10 bus 11
>>> vmx0 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:a9:8f
>>> ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
>>> pci12 at ppb11 bus 12
>>> vmx1 at pci12 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:3a:8a
>>> ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
>>> pci13 at ppb12 bus 13
>>> vmx2 at pci13 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:6d:7d
>>> ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
>>> pci14 at ppb13 bus 14
>>> vmx3 at pci14 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:b5:57
>>> ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
>>> pci15 at ppb14 bus 15
>>> ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
>>> pci16 at ppb15 bus 16
>>> ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
>>> pci17 at ppb16 bus 17
>>> ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
>>> pci18 at ppb17 bus 18
>>> ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
>>> pci19 at ppb18 bus 19
>>> vmx4 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:62:2e
>>> ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
>>> pci20 at ppb19 bus 20
>>> vmx5 at pci20 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:e5:4e
>>> ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
>>> pci21 at ppb20 bus 21
>>> vmx6 at pci21 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:b3:41
>>> ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
>>> pci22 at ppb21 bus 22
>>> vmx7 at pci22 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:98:e0
>>> ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
>>> pci23 at ppb22 bus 23
>>> ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
>>> pci24 at ppb23 bus 24
>>> ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
>>> pci25 at ppb24 bus 25
>>> ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
>>> pci26 at ppb25 bus 26
>>> ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
>>> pci27 at ppb26 bus 27
>>> vmx8 at pci27 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 17,
address 00:50:56:92:04:05
>>> ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
>>> pci28 at ppb27 bus 28
>>> vmx9 at pci28 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 17,
address 00:50:56:92:2c:ed
>>> ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
>>> pci29 at ppb28 bus 29
>>> ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
>>> pci30 at ppb29 bus 30
>>> ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
>>> pci31 at ppb30 bus 31
>>> ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
>>> pci32 at ppb31 bus 32
>>> ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
>>> pci33 at ppb32 bus 33
>>> ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
>>> pci34 at ppb33 bus 34
>>> isa0 at pcib0
>>> isadma0 at isa0
>>> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
>>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>>> pckbd0 at pckbc0 (kbd slot)
>>> wskbd0 at pckbd0: console keyboard, using wsdisplay0
>>> pms0 at pckbc0 (aux slot)
>>> wsmouse0 at pms0 mux 0
>>> pcppi0 at isa0 port 0x61
>>> spkr0 at pcppi0
>>> lpt0 at isa0 port 0x378/4 irq 7
>>> vscsi0 at root
>>> scsibus3 at vscsi0: 256 targets
>>> softraid0 at root
>>> scsibus4 at softraid0: 256 targets
>>> root on sd0a (30d13733f6764d22.a) swap on sd0b dump on sd0b
>>> WARNING: / was not properly unmounted
>>> carp: pfsync0 demoted group carp by 32 to 160 (pfsync init)
>>> carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init)
>>> carp: pfsync0 demoted group carp by 1 to 161 (pfsync bulk start)
>>> carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start)
>>> carp239: state transition: BACKUP -> MASTER
>>> carp241: state transition: BACKUP -> MASTER
>>> carp243: state transition: BACKUP -> MASTER
>>> carp: pfsync0 demoted group carp by -1 to 32 (pfsync bulk done)
>>> carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk done)
>>> carp: pfsync0 demoted group carp by -32 to 0 (pfsync init)
>>> carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)
>>> carp238: state transition: BACKUP -> MASTER
>>> carp240: state transition: BACKUP -> MASTER
>>> carp242: state transition: BACKUP -> MASTER
>>> carp244: state transition: BACKUP -> MASTER
>>> carp302: state transition: BACKUP -> MASTER
>>> carp238: state transition: MASTER -> BACKUP
>>> carp240: state transition: MASTER -> BACKUP
>>> carp242: state transition: MASTER -> BACKUP
>>> carp302: state transition: MASTER -> BACKUP
>>> carp244: state transition: MASTER -> BACKUP
