I was unable to trigger this with
OpenBSD 5.9 (GENERIC.MP) #1869: Thu Feb 4 09:50:59 MST 2016
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
//mxb
> On 5 feb. 2016, at 19:12, mxb <m...@alumni.chalmers.se> wrote:
>
>
> Any one from @devs have time to pick it up?
>
> This is a new env. , so I have time to investigate.
> Access can be provided on need bases.
>
> //mxb
>
>> On 4 feb. 2016, at 15:46, mxb <m...@alumni.chalmers.se> wrote:
>>
>> Found it in dmesg buffer:
>>
>> Stopped at Debugger+0x9: leave
>> RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS
PANIC!
>> IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS,
TOO.
>> DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
>>
>> ddb{0}> trace
>> Debugger() at Debugger+0x9
>> panic() at panic+0xfe
>> mtx_enter() at mtx_enter+0x60
>> sofree() at sofree+0xa0
>> in_pcbdetach() at in_pcbdetach+0x40
>> tcp_close() at tcp_close+0xad
>> tcp_timer_2msl() at tcp_timer_2msl+0x90
>> softclock() at softclock+0x315
>> softintr_dispatch() at softintr_dispatch+0x8b
>> Xsoftclock() at Xsoftclock+0x1f
>> --- interrupt ---
>> (null)() at 0x8
>> end of kernel
>> end trace frame: 0x1102000000000001, count: -11
>>
>> ddb{0}> show registers
>> rdi 0x1
>> rsi 0x282
>> rbp 0xffff8000221c68f8
>> rbx 0xffffffff813285e0 mtx_enter+0x60
>> rdx 0
>> rcx 0xffffffff8188c640 cpu_info_primary
>> rax 0x1
>> r8 0xffff8000221c6818
>> r9 0x1
>> r10 0
>> r11 0xffff8000221c66b0
>> r12 0x100
>> r13 0xffff8000221c6908
>> r14 0xffffff011d088010
>> r15 0
>> rip 0xffffffff81343b09 Debugger+0x9
>> cs 0x8
>> rflags 0x286
>> rsp 0xffff8000221c68e8
>> ss 0x10
>> Debugger+0x9: leave
>>
>> ddb{0}> ps
>> PID PPID PGRP UID S FLAGS WAIT COMMAND
>> 19921 12801 19921 0 3 0x83 poll systat
>> 12801 569 12801 0 3 0x8b pause ksh
>> 569 23137 569 0 3 0x92 select sshd
>> 13678 1 13678 0 3 0x83 ttyin getty
>> 9776 1 9776 0 3 0x83 ttyin getty
>> 3392 1 3392 0 3 0x83 ttyin getty
>> 24230 1 24230 0 3 0x83 ttyin getty
>> 14469 1 14469 0 3 0x83 ttyin getty
>> 29209 1 29209 0 3 0x80 poll cron
>> 4796 4087 4087 95 3 0x90 kqread smtpd
>> 269 4087 4087 95 3 0x90 kqread smtpd
>> 28144 4087 4087 95 3 0x90 kqread smtpd
>> 13626 4087 4087 95 3 0x90 kqread smtpd
>> 4756 4087 4087 95 3 0x90 kqread smtpd
>> 23276 4087 4087 103 3 0x90 kqread smtpd
>> 4087 1 4087 0 3 0x80 kqread smtpd
>> 29277 28344 28344 89 3 0x90 kqread relayd
>> 15361 28344 28344 89 3 0x90 kqread relayd
>> 28344 26987 28344 89 3 0x90 kqread relayd
>> *18801 1528 1528 89 7 0x10 relayd
>> 15066 1528 1528 89 3 0x90 kqread relayd
>> 1528 26987 1528 89 3 0x90 kqread relayd
>> 14013 26987 14013 89 3 0x90 kqread relayd
>> 25397 26987 25397 89 3 0x90 kqread relayd
>> 26987 1 26987 0 3 0x80 kqread relayd
>> 23945 11110 11110 85 3 0x90 kqread ospfd
>> 12948 11110 11110 85 3 0x90 kqread ospfd
>> 11110 1 11110 0 3 0x80 kqread ospfd
>> 23137 1 23137 0 3 0x80 select sshd
>> 10031 27507 981 83 3 0x90 poll ntpd
>> 27507 981 981 83 3 0x90 poll ntpd
>> 981 1 981 0 3 0x80 poll ntpd
>> 12220 25415 25415 74 3 0x90 bpf pflogd
>> 25415 1 25415 0 3 0x80 netio pflogd
>> 3275 32486 32486 73 3 0x90 kqread syslogd
>> 32486 1 32486 0 3 0x80 netio syslogd
>> 14861 0 0 0 3 0x14200 pgzero zerothread
>> 24670 0 0 0 3 0x14200 aiodoned aiodoned
>> 29165 0 0 0 3 0x14200 syncer update
>> 27875 0 0 0 3 0x14200 cleaner cleaner
>> 645 0 0 0 3 0x14200 reaper reaper
>> 17692 0 0 0 3 0x14200 pgdaemon pagedaemon
>> 766 0 0 0 3 0x14200 bored crypto
>> 4241 0 0 0 3 0x14200 pftm pfpurge
>> 9257 0 0 0 3 0x40014200 acpi0 acpi0
>> 25569 0 0 0 7 0x40014200 idle1
>> 7980 0 0 0 3 0x14200 bored sensors
>> 3321 0 0 0 3 0x14200 bored softnet
>> 4749 0 0 0 3 0x14200 bored systqmp
>> 16058 0 0 0 3 0x14200 bored systq
>> 15954 0 0 0 3 0x40014200 idle0
>> 1 0 1 0 3 0x82 wait init
>> 0 -1 0 0 3 0x10200 scheduler swapper
>>
>>> On 4 feb. 2016, at 13:49, mxb <m...@alumni.chalmers.se> wrote:
>>>
>>>
>>> I was able to re-produce this panic with similar stack trace.
>>> Unfortunately 'trace/show regs/ps' are not in txt format, but are
screenshots.
>>>
>>> //mxb
>>>
>>>> On 4 feb. 2016, at 12:42, mxb <m...@alumni.chalmers.se> wrote:
>>>>
>>>>
>>>> Hey,
>>>> see those again on 5.8-STABLE.
>>>>
>>>> This is a 2-node CARP setup within VMWare ESX.
>>>> Both machines are rebooting after this and it happens quite often.
>>>>
>>>> Any ideas?
>>>>
>>>> panic: mtx_enter: locking against myself
>>>> Starting stack trace...
>>>> panic() at panic+0x10b
>>>> mtx_enter() at mtx_enter+0x60
>>>> sofree() at sofree+0xa0
>>>> in_pcbdetach() at in_pcbdetach+0x40
>>>> tcp_close() at tcp_close+0xad
>>>> tcp_timer_2msl() at tcp_timer_2msl+0x90
>>>> softclock() at softclock+0x315
>>>> softintr_dispatch() at softintr_dispatch+0x8b
>>>> Xsoftclock() at Xsoftclock+0x1f
>>>> --- interrupt ---
>>>> (null)() at 0x8
>>>> (null)() at 0xffffff0118c50f90
>>>> end trace frame: 0x0, count: 246
>>>> End of stack trace.
>>>> syncing disks... panic: assertwaitok: non-zero mutex count: 1
>>>> Starting stack trace...
>>>> panic() at panic+0x10b
>>>> assertwaitok() at assertwaitok+0x52
>>>> bufq_wait() at bufq_wait+0x2d
>>>> bwrite() at bwrite+0xfe
>>>> VOP_BWRITE() at VOP_BWRITE+0x38
>>>> ffs_fsync() at ffs_fsync+0x13f
>>>> VOP_FSYNC() at VOP_FSYNC+0x3c
>>>> ffs_sync() at ffs_sync+0xc4
>>>> sys_sync() at sys_sync+0x87
>>>> vfs_syncwait() at vfs_syncwait+0x50
>>>> vfs_shutdown() at panic: mtx_enter: locking against myself
>>>> Faulted in traceback, aborting…
>>>>
>>>> OpenBSD 5.8-stable (GENERIC.MP) #0: Thu Oct 22 18:52:06 CEST 2015
>>>> root@prdlba0001:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>>> real mem = 4278124544 (4079MB)
>>>> avail mem = 4144574464 (3952MB)
>>>> mpath0 at root
>>>> scsibus0 at mpath0: 256 targets
>>>> mainbus0 at root
>>>> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (364 entries)
>>>> bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014
>>>> bios0: VMware, Inc. VMware Virtual Platform
>>>> acpi0 at bios0: rev 2
>>>> acpi0: sleep states S0 S1 S4 S5
>>>> acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
>>>> acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3)
S3F0(S3) S4F0(S3) S5F0(S3) S6F0(S3) S7F0(S3) S8F0(S3) S9F0(S3) S10F(S3)
S11F(S3) S12F(S3) S13F(S3) [...]
>>>> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>>>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>>>> cpu0 at mainbus0: apid 0 (boot processor)
>>>> cpu0: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2400.20 MHz
>>>> cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,PERF,I
TSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
>>>> cpu0: 256KB 64b/line 8-way L2 cache
>>>> cpu0: smt 0, core 0, package 0
>>>> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
>>>> cpu0: apic clock running at 65MHz
>>>> cpu1 at mainbus0: apid 1 (application processor)
>>>> cpu1: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz, 2399.74 MHz
>>>> cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,
x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,PERF,I
TSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
>>>> cpu1: 256KB 64b/line 8-way L2 cache
>>>> cpu1: smt 0, core 1, package 0
>>>> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
>>>> acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127
>>>> acpihpet0 at acpi0: 14318179 Hz
>>>> acpiprt0 at acpi0: bus 0 (PCI0)
>>>> acpicpu0 at acpi0: C1(@1 halt!)
>>>> acpicpu1 at acpi0: C1(@1 halt!)
>>>> acpibat0 at acpi0: BAT1 not present
>>>> acpibat1 at acpi0: BAT2 not present
>>>> acpiac0 at acpi0: AC unit online
>>>> acpibtn0 at acpi0: SLPB
>>>> acpibtn1 at acpi0: LID_
>>>> pvbus0 at mainbus0: VMware
>>>> vmt0 at pvbus0
>>>> pci0 at mainbus0 bus 0
>>>> pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
>>>> ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
>>>> pci1 at ppb0 bus 1
>>>> pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
>>>> pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
>>>> pciide0: channel 0 disabled (no drives)
>>>> pciide0: channel 1 disabled (no drives)
>>>> piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus
disabled
>>>> "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
>>>> vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
>>>> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
>>>> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
>>>> ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
>>>> pci2 at ppb1 bus 2
>>>> ahci0 at pci2 dev 1 function 0 "VMware AHCI" rev 0x00: apic 2 int 19,
AHCI 1.3
>>>> ahci0: port 0: 6.0Gb/s
>>>> scsibus1 at ahci0: 32 targets
>>>> cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware SATA CD00, 1.00> ATAPI
5/cdrom removable
>>>> ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
>>>> pci3 at ppb2 bus 3
>>>> vmwpvs0 at pci3 dev 0 function 0 "VMware PVSCSI" rev 0x02: apic 2 int 18
>>>> scsibus2 at vmwpvs0: 16 targets
>>>> sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct
fixed
>>>> sd0: 16384MB, 512 bytes/sector, 33554432 sectors
>>>> ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
>>>> pci4 at ppb3 bus 4
>>>> ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
>>>> pci5 at ppb4 bus 5
>>>> ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
>>>> pci6 at ppb5 bus 6
>>>> ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
>>>> pci7 at ppb6 bus 7
>>>> ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
>>>> pci8 at ppb7 bus 8
>>>> ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
>>>> pci9 at ppb8 bus 9
>>>> ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
>>>> pci10 at ppb9 bus 10
>>>> ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
>>>> pci11 at ppb10 bus 11
>>>> vmx0 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:a9:8f
>>>> ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
>>>> pci12 at ppb11 bus 12
>>>> vmx1 at pci12 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:3a:8a
>>>> ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
>>>> pci13 at ppb12 bus 13
>>>> vmx2 at pci13 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:6d:7d
>>>> ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
>>>> pci14 at ppb13 bus 14
>>>> vmx3 at pci14 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 19,
address 00:50:56:92:b5:57
>>>> ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
>>>> pci15 at ppb14 bus 15
>>>> ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
>>>> pci16 at ppb15 bus 16
>>>> ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
>>>> pci17 at ppb16 bus 17
>>>> ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
>>>> pci18 at ppb17 bus 18
>>>> ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
>>>> pci19 at ppb18 bus 19
>>>> vmx4 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:62:2e
>>>> ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
>>>> pci20 at ppb19 bus 20
>>>> vmx5 at pci20 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:e5:4e
>>>> ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
>>>> pci21 at ppb20 bus 21
>>>> vmx6 at pci21 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:b3:41
>>>> ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
>>>> pci22 at ppb21 bus 22
>>>> vmx7 at pci22 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 16,
address 00:50:56:92:98:e0
>>>> ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
>>>> pci23 at ppb22 bus 23
>>>> ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
>>>> pci24 at ppb23 bus 24
>>>> ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
>>>> pci25 at ppb24 bus 25
>>>> ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
>>>> pci26 at ppb25 bus 26
>>>> ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
>>>> pci27 at ppb26 bus 27
>>>> vmx8 at pci27 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 17,
address 00:50:56:92:04:05
>>>> ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
>>>> pci28 at ppb27 bus 28
>>>> vmx9 at pci28 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 2 int 17,
address 00:50:56:92:2c:ed
>>>> ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
>>>> pci29 at ppb28 bus 29
>>>> ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
>>>> pci30 at ppb29 bus 30
>>>> ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
>>>> pci31 at ppb30 bus 31
>>>> ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
>>>> pci32 at ppb31 bus 32
>>>> ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
>>>> pci33 at ppb32 bus 33
>>>> ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
>>>> pci34 at ppb33 bus 34
>>>> isa0 at pcib0
>>>> isadma0 at isa0
>>>> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
>>>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>>>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>>>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>>>> pckbd0 at pckbc0 (kbd slot)
>>>> wskbd0 at pckbd0: console keyboard, using wsdisplay0
>>>> pms0 at pckbc0 (aux slot)
>>>> wsmouse0 at pms0 mux 0
>>>> pcppi0 at isa0 port 0x61
>>>> spkr0 at pcppi0
>>>> lpt0 at isa0 port 0x378/4 irq 7
>>>> vscsi0 at root
>>>> scsibus3 at vscsi0: 256 targets
>>>> softraid0 at root
>>>> scsibus4 at softraid0: 256 targets
>>>> root on sd0a (30d13733f6764d22.a) swap on sd0b dump on sd0b
>>>> WARNING: / was not properly unmounted
>>>> carp: pfsync0 demoted group carp by 32 to 160 (pfsync init)
>>>> carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init)
>>>> carp: pfsync0 demoted group carp by 1 to 161 (pfsync bulk start)
>>>> carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start)
>>>> carp239: state transition: BACKUP -> MASTER
>>>> carp241: state transition: BACKUP -> MASTER
>>>> carp243: state transition: BACKUP -> MASTER
>>>> carp: pfsync0 demoted group carp by -1 to 32 (pfsync bulk done)
>>>> carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk done)
>>>> carp: pfsync0 demoted group carp by -32 to 0 (pfsync init)
>>>> carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)
>>>> carp238: state transition: BACKUP -> MASTER
>>>> carp240: state transition: BACKUP -> MASTER
>>>> carp242: state transition: BACKUP -> MASTER
>>>> carp244: state transition: BACKUP -> MASTER
>>>> carp302: state transition: BACKUP -> MASTER
>>>> carp238: state transition: MASTER -> BACKUP
>>>> carp240: state transition: MASTER -> BACKUP
>>>> carp242: state transition: MASTER -> BACKUP
>>>> carp302: state transition: MASTER -> BACKUP
>>>> carp244: state transition: MASTER -> BACKUP
