Hi,
We're running several OpenBSD 4.6 VMs under VMWare ESX 3.0.2 52542, and
are seeing random panics with very similar panic messages once every
fortnight or so.
The kernel is GENERIC, but with vmt0 (VMWare tools driver) uncommented.
If anyone could shed any light on this, I would be very grateful.
A couple of examples...
VM 3:
----- panic message
panic: pool_do_get(sigapl): free list modified: page 0xd516e000; item addr
0xd516e008; offset 0x0=0x206d656c
Stopped at Debugger+0x4: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb> trace
Debugger(f13f,d7589e1c,d76f4000,d516e008,d08905c0) at Debugger+0x4
panic(d0716c20,d07150c4,d516e000,d516e008,0) at panic+0x55
pool_do_get(d08905c0,2,0,d5301f30,d51922c4) at pool_do_get+0x2e3
pool_get(d08905c0,2,d51c44b4,d5301f30,d51922c4) at pool_get+0x46
sigactsinit(d51c6578,d5300f30,18,1,0) at sigactsinit+0x14
fork1(d51c6578,14,1,0,0) at fork1+0x6ee
sys_fork(d51c6578,d7589f68,d7589f58,18,3aa99) at sys_fork+0x38
syscall() at syscall+0x24e
--- syscall (number 2) ---
0xcf4541d:
ddb> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
14151 4050 4050 521 2 0x40100 exim
17995 4050 4050 521 2 0x40100 exim
13802 27015 4050 521 2 0x40100 exim
7334 1739 4050 521 3 0x40180 select exim
16807 4050 4050 521 2 0xc0100 exim
32328 2426 4050 521 2 0x40100 exim
23715 4050 4050 521 2 0xc0100 exim
4853 4050 4050 521 2 0xc0100 exim
21026 4050 4050 521 2 0xc0100 exim
15345 4050 4050 521 2 0x40100 exim
16650 1 4050 521 2 0x40100 exim
2831 20287 4050 521 2 0x40100 exim
16786 1833 4050 521 3 0x40180 select exim
139 4050 4050 521 2 0xc0100 exim
22188 1 4050 521 2 0x40100 exim
13801 4050 4050 521 2 0xc0180 exim
5803 7246 4050 521 2 0x40100 exim
22554 1 4050 521 2 0x44100 exim
30675 1 4050 521 2 0x44100 exim
32000 1 4050 521 2 0x44100 exim
14680 28431 4050 521 2 0x40180 exim
932 1 4050 521 2 0x44100 exim
23412 26226 4050 521 2 0x40100 exim
24507 1 4050 0 2 0x44100 exim
3532 19041 4050 521 3 0x40180 select exim
10835 1 4050 0 2 0x44100 exim
28575 3999 4050 521 2 0x40100 exim
12055 28537 4050 521 2 0x40100 exim
1739 1 4050 0 3 0x44180 select exim
6667 1 4050 521 2 0x44100 exim
14700 18917 4050 521 2 0x40100 exim
6301 24603 4050 521 2 0x40100 exim
24603 1 4050 0 3 0x44180 select exim
18917 1 4050 0 3 0x44180 wait exim
23897 6022 4050 521 2 0x40100 exim
9803 1 4050 521 2 0x44100 exim
7246 1 4050 0 3 0x44180 select exim
5146 4050 4050 521 2 0xc0100 exim
28196 4050 4050 521 2 0xc0100 exim
12560 4050 4050 521 2 0xc0100 exim
28251 4050 4050 521 2 0xc0100 exim
2426 4050 4050 521 2 0xc0100 exim
19190 4050 4050 521 2 0xc0100 exim
17179 4050 4050 521 2 0xc0180 exim
27015 4050 4050 521 2 0xc0100 exim
14161 4050 4050 521 2 0xc0100 exim
32502 4050 4050 521 2 0xc0100 exim
9113 4050 4050 521 2 0xc0100 exim
795 4050 4050 521 2 0xc0100 exim
18497 4050 4050 521 2 0xc0100 exim
3999 1 4050 0 3 0x44180 wait exim
6022 1 4050 0 3 0x44180 wait exim
28225 4050 4050 521 2 0xc0100 exim
20021 4050 4050 521 2 0xc0100 exim
18778 4050 4050 521 2 0xc0100 exim
4095 4050 4050 521 2 0xc0100 exim
23081 4050 4050 521 2 0xc0100 exim
1784 4050 4050 521 2 0xc0100 exim
4677 4050 4050 521 2 0xc0100 exim
900 4050 4050 521 2 0xc0100 exim
23499 4050 4050 521 2 0xc0100 exim
28431 1 4050 0 3 0x44180 select exim
24179 4050 4050 521 2 0xc0100 exim
1833 1 4050 0 3 0x44180 select exim
28537 1 4050 0 3 0x44180 select exim
18624 1 4050 0 2 0x44100 exim
26226 1 4050 0 3 0x44180 select exim
19041 1 4050 0 3 0x44180 select exim
13092 1 4050 0 2 0x44100 exim
20287 1 4050 0 3 0x44180 select exim
* 4050 1 4050 521 7 0x40100 exim
5491 1 5491 0 3 0x4082 ttyin getty
14541 1 14541 0 3 0x4082 ttyin getty
8905 1 8905 0 3 0x4082 ttyin getty
14080 1 14080 0 3 0x4082 ttyin getty
30717 1 30717 0 3 0x4082 ttyin getty
28074 1 28074 0 3 0x4082 ttyin getty
19681 1 19681 0 3 0x80 select cron
5959 1 28480 0 3 0x80 select snmpd
3361 1 3361 0 3 0x80 select sshd
31996 1 31996 0 3 0x80 kqread ifstated
28282 28853 28853 85 3 0x180 kqread ospfd
13872 28853 28853 85 3 0x180 kqread ospfd
28853 1 28853 0 3 0x80 kqread ospfd
17545 28861 21933 83 3 0x180 poll ntpd
28861 21933 21933 83 3 0x180 poll ntpd
21933 1 21933 0 3 0x80 poll ntpd
32703 29509 29509 73 3 0x180 poll syslogd
29509 1 29509 0 3 0x88 netio syslogd
12 0 0 0 3 0x100200 bored crypto
11 0 0 0 3 0x100200 aiodoned aiodoned
10 0 0 0 3 0x100200 syncer update
9 0 0 0 3 0x100200 cleaner cleaner
8 0 0 0 3 0x100200 reaper reaper
7 0 0 0 3 0x100200 pgdaemon pagedaemon
6 0 0 0 3 0x100200 pftm pfpurge
5 0 0 0 3 0x100200 apmev apm0
4 0 0 0 3 0x100200 bored syswq
3 0 0 0 3 0x40100200 idle0
2 0 0 0 3 0x100200 kmalloc kmthread
1 0 1 0 3 0x4080 wait init
0 -1 0 0 3 0x80200 scheduler swapper
----- uname
OpenBSD vm-03.tld 4.6 GENERIC-VMWARE#0 i386
----- dmesg
OpenBSD 4.6 (GENERIC-VMWARE) #0: Mon Mar 29 10:31:14 BST 2010
r...@build-01.tld:/usr/src/sys/arch/i386/compile/GENERIC-VMWARE
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.82 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS
real mem = 267939840 (255MB)
avail mem = 250261504 (238MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 04/17/06, BIOS32 rev. 0 @ 0xfd880, SMBIOS
rev. 2.31 @ 0xe0010 (45 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 04/17/2006
bios0: VMware, Inc. VMware Virtual Platform
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
acpi at bios0 function 0x0 not configured
pcibios0 at bios0: rev 2.1 @ 0xfd880/0x780
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdf30/176 (9 entries)
pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1a00! 0xca000/0x1000 0xdc000/0x4000!
0xe0000/0x4000!
vmt0 at mainbus0
cpu0 at mainbus0: (uniprocessor)
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
piixpcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <NECVMWar, VMware IDE CDR00, 1.00> ATAPI 5/cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled
vga1 at pci0 dev 15 function 0 "VMware Virtual SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
mpi0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: irq 9
scsibus1 at mpi0: 16 targets, initiator 7
sd0 at scsibus1 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed
sd0: 4096MB, 512 bytes/sec, 8388608 sec total
mpi0: target 0 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1
vic0 at pci0 dev 17 function 0 "AMD 79c970 PCnet-PCI" rev 0x10: irq 11, address
00:0c:29:6a:be:43
isa0 at piixpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask e765 netmask ef65 ttymask ffff
mtrr: CPU supports MTRRs but not enabled
softraid0 at root
root on sd0a swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
vmt0: hostname set to vm-03.tld
-----
VM 4:
----- panic message
panic: pool_do_get(mbpl): free list modified: page 0xd5130000; item addr
0xd5130100; offset 0x0=0x538c0883
Stopped at Debugger+0x4: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb> trace
Debugger(d52b0400,d0894ff8,d7540de0,d5130100,d0894f80) at Debugger+0x4
panic(d0716c20,d0718a84,d5130000,d5130100,0) at panic+0x55
pool_do_get(d0894f80,0,d7540e00,d036478f,d0894f80) at
pool_do_get+0x2e3
pool_get(d0894f80,0,d7540e50,d033f3bb) at pool_get+0x46
m_gethdr(1,1,e62d082,d52b0400) at m_gethdr+0x2d
vic_alloc_mbuf(d0f5d800,d0fdec00,800,10e) at vic_alloc_mbuf+0x18
vic_rx_fill(d0f5d800,0,d52b0400,d0f5d5c8,edcd754) at vic_rx_fill+0x5c
vic_rx_proc(d0f5d800,0,f400f50,ef00040) at vic_rx_proc+0x14a
vic_intr(d0f5d800) at vic_intr+0x15
Xrecurse_legacy11() at Xrecurse_legacy11+0xb1
--- interrupt ---
apm_cpu_idle(d0356151,d753f000,d09eae78,d0200928,d0941920) at
apm_cpu_idle+0x2c
cpu_idle_cycle(d0941920) at cpu_idle_cycle+0xc
Bad frame pointer: 0xd09eae78
ddb> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
24516 1 24516 0 3 0x4082 ttyin getty
10503 1 10503 0 3 0x4082 ttyin getty
23516 1 23516 0 3 0x4082 ttyin getty
5895 1 5895 0 3 0x4082 ttyin getty
13717 1 13717 0 3 0x4082 ttyin getty
5304 1 5304 0 3 0x4082 ttyin getty
25820 1 25820 0 3 0x80 select cron
26526 1 26526 559 3 0x180 poll radiusd
25614 1 29903 0 3 0x80 select snmpd
12249 1 12249 0 3 0x80 select sshd
27175 1 15615 521 3 0x44182 select exim
15501 1 15501 0 3 0x80 kqread ifstated
25071 10906 10906 85 3 0x180 kqread ospfd
13925 10906 10906 85 3 0x180 kqread ospfd
10906 1 10906 0 3 0x80 kqread ospfd
6739 3910 21852 83 3 0x180 poll ntpd
3910 21852 21852 83 3 0x180 poll ntpd
21852 1 21852 0 3 0x80 poll ntpd
5130 7129 7129 73 3 0x180 poll syslogd
7129 1 7129 0 3 0x88 netio syslogd
12 0 0 0 3 0x100200 bored crypto
11 0 0 0 3 0x100200 aiodoned aiodoned
10 0 0 0 3 0x100200 syncer update
9 0 0 0 3 0x100200 cleaner cleaner
8 0 0 0 3 0x100200 reaper reaper
7 0 0 0 3 0x100200 pgdaemon pagedaemon
6 0 0 0 3 0x100200 pftm pfpurge
5 0 0 0 3 0x100200 apmev apm0
4 0 0 0 3 0x100200 bored syswq
* 3 0 0 0 7 0x40100200 idle0
2 0 0 0 3 0x100200 kmalloc kmthread
1 0 1 0 3 0x4080 wait init
0 -1 0 0 3 0x80200 scheduler swapper
----- uname
OpenBSD vm-04.tld 4.6 GENERIC-VMWARE#0 i386
----- dmesg
OpenBSD 4.6 (GENERIC-VMWARE) #0: Mon Mar 29 10:31:14 BST 2010
r...@build-01.tld:/usr/src/sys/arch/i386/compile/GENERIC-VMWARE
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.79 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS
real mem = 267939840 (255MB)
avail mem = 250261504 (238MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 04/17/06, BIOS32 rev. 0 @ 0xfd880, SMBIOS
rev. 2.31 @ 0xe0010 (45 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 04/17/2006
bios0: VMware, Inc. VMware Virtual Platform
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
acpi at bios0 function 0x0 not configured
pcibios0 at bios0: rev 2.1 @ 0xfd880/0x780
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfdf30/176 (9 entries)
pcibios0: PCI Interrupt Router at 000:07:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCI bus #1 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1a00! 0xca000/0x1000 0xdc000/0x4000!
0xe0000/0x4000!
vmt0 at mainbus0
cpu0 at mainbus0: (uniprocessor)
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
piixpcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <NECVMWar, VMware IDE CDR00, 1.00> ATAPI 5/cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled
vga1 at pci0 dev 15 function 0 "VMware Virtual SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
mpi0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: irq 9
scsibus1 at mpi0: 16 targets, initiator 7
sd0 at scsibus1 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed
sd0: 4096MB, 512 bytes/sec, 8388608 sec total
mpi0: target 0 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1
vic0 at pci0 dev 17 function 0 "AMD 79c970 PCnet-PCI" rev 0x10: irq 11, address
00:0c:29:c1:0f:e9
isa0 at piixpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask e765 netmask ef65 ttymask ffff
mtrr: CPU supports MTRRs but not enabled
softraid0 at root
root on sd0a swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
vmt0: hostname set to vm-04.tld
-----
Thanks in advance!
Alex
