Hello all,
I have a bit of a problem here, which I figured you might be able to shed
some light on.
Setup:
2 x obsd 3.8 (+patches) machines running pf/pfsync/carp/ftp-proxy. Using 4
carp interfaces per machine,
plus an if for pfsync. Hardware used is a couple of ibm x306, each with an
intel quad-gigabit nic and two onboard
intel gigabit nics.
During my vacation the primary firewall panic'ed, and for some reason the
secondary fw didn't take over,
the crasched firewall was rebooted and seemed to work ok. My coworker did a
bit of research on the cause
of the problem, and decided to try and increase maxclusters (
kern.maxclusters: 6144 -> 15000).
~3 days later the primary firewall deciced to take another break, but this
time around the secondary took
over ok.
Now, about 7 weeks later, the primary panic'ed again (secondary taking over
ok),
attached below is some output from ddb and boot, (I couldn't find anything
relevant in the logfiles)
Any hints/comments/etc on this is most welcome.
regards,
/Anders
OpenBSD/i386 (fw0-host.my.domain) (ttyC0)
login: panic: pool_get(mclpl): free list modified: magic=19beab21; page
0xd698d0
00; item addr 0xd698d000
Stopped at Debugger+0x4: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb> ps
PID PPID PGRP UUID S FLAGS WAIT COMMAND
25520 16911 25520 71 3 0x4184 select ftp-proxy
10352 16911 10352 71 3 0x4184 select ftp-proxy
6432 16911 6432 71 3 0x4184 select ftp-proxy
21010 23754 23753 73 3 0x184 poll syslogd
23754 1 23754 0 3 0x84 netio syslogd
17066 16911 17066 71 3 0x4184 select ftp-proxy
14347 16911 14347 71 3 0x4184 select ftp-proxy
15416 16911 15416 71 3 0x4184 select ftp-proxy
1806 1 1806 0 3 0x4086 ttyin getty
25071 1 25071 0 3 0x40184 select sendmail
31017 1 31017 0 3 0x4086 ttyin getty
18299 1 18299 0 3 0x4086 ttyin getty
3480 1 3480 0 3 0x4086 ttyin getty
23739 1 23739 0 3 0x4086 ttyin getty
805 1 805 0 3 0x84 select cron
25276 1 25276 0 3 0x84 select sshd
16911 1 16911 0 3 0x184 select inetd
12567 8424 8424 83 3 0x184 poll ntpd
8424 1 8424 0 3 0x84 poll ntpd
21663 27056 27056 74 3 0x184 bpf pflogd
27056 1 27056 0 3 0x84 netio pflogd
13 0 0 0 3 0x100204 crypto_wa crypto
12 0 0 0 3 0x100204 aiodoned aiodoned
11 0 0 0 3 0x100204 syncer update
10 0 0 0 3 0x100204 cleaner cleaner
9 0 0 0 3 0x100204 reaper reaper
8 0 0 0 3 0x100204 pgdaemon pagedaemon
7 0 0 0 3 0x100204 pftm pfpurge
6 0 0 0 3 0x100204 usbevt usb2
5 0 0 0 3 0x100204 usbevt usb1
4 0 0 0 3 0x100204 usbtsk usbtask
3 0 0 0 3 0x100204 usbevt usb0
2 0 0 0 3 0x100204 kmalloc kmthread
1 0 1 0 3 0x4084 wait init
0 -1 0 0 3 0x80204 scheduler swapper
ddb> trace
Debugger(5e000, 14000201,6820285e,d698d000,d05d27c0) at Debugger+0x4
panic(d04f6c40,d04f8c09,19beab21,d698d000,d698d000) at panic+0x63
pool_get(d05d27c0,0,d06f1dcc,d0254fcf,d0f67830) at pool_get+0x315
em_get_buf(23,d0f67800,0,d10505ee) at em_get_buf+0x176
em_process_receive_interrupts(d0f67800,fffffff8,d0101f50,4,d06f1e44) at
em_proc
ess_receive_interrupts+0x23a
em_intr(d0f67800) at em_intr+0x93
Xrecurse_legacy11() at Xrecurse_legacy11+0x8a
--- interrupt ---
apm_cpu_idle(b0,d05ccec0,d05ccd40,7fffffff,d021ae67) at apm_cpu_idle+0x42
idle_loop(80058,10,0,0,80000000) at idle_loop+0x5
bpendtsleep(d05ccd40,4,d050e4b1,0,0,d0307c16,8,286) at bpendtsleep
uvm_scheduler(d05ccd3c,3,0,d04c7492,1ff70000) at uvm_scheduler+0x6b
check_console(0,0,0,0,0) at check_console
ddb> boot dump
panic: pool_get(mclpl): free list modified: magic=19beab21; page 0xd698d000;
ite
m addr 0xd698d000
Stopped at Debugger+0x4: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb>
Using drive 0, partition 3.
Loading...
probing: pc0 com0 mem[622K 510M a20=on]
disk: fd hd0+ hd1+
>> OpenBSD/i386 BOOT 2.10
boot>
booting hd0a:/bsd: 4804448+939504 [52+247296+228813]=0x5eeac8
entry point at 0x100120
[ using 476536 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2005 OpenBSD. All rights reserved.
http://www.OpenBSD.org
OpenBSD 3.8 (GENERIC) #0: Mon Feb 6 11:04:29 CET 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
real mem = 535846912 (523288K)
avail mem = 482033664 (470736K)
using 4278 buffers containing 26894336 bytes (26264K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(26) BIOS, date 06/16/05, BIOS32 rev. 0 @ 0xfd5b6
pcibios0 at bios0: rev 2.1 @ 0xfd520/0xae0
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfde80/352 (20 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371FB ISA" rev 0x00)
pcibios0: PCU bus #4 is the last bus
bios0: ROM list: 0xc0000/0x9000 0xc9000/0x1000 0xca000/0x1000
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82875P Host" rev 0x02
ppb0 at pci0 dev 3 function 0 "Intel 82875P PCI-CSA" rev 0x02
pci1 at ppb0 bus 1
em0 at pci1 dev 1 function 0 "Intel PRO/1000CT (82547GI)" rev 0x00: irq 5,
addre
ss: 00:14:5e:41:7a:d2
ppb1 at pci0 dev 28 function 0 "Intel 6300ESB PCIX" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci2 dev 1 function 0 "IBM PCIX-PCIX" rev 0x02
pci3 at ppbb2 bus 3
em1 at pci3 dev 4 function 0 "Intel PRO/1000MT QP (82546EB)" rev 0x01: irq
11, a
ddress: 00:04:23:c4:5b:64
em2 at pci3 dev 4 function 1 "Intel PRO/1000MT QP (82546EB)" rev 0x01: irq
11, a
ddress: 00:04:23:c4:5b:65
em3 at pci3 dev 6 function 0 "Intel PRO/1000MT QP (82546EB)" rev 0x01: irq
11, a
ddress: 00:04:23:c4:5b:66
em4 at pci3 dev 6 function 1 "Intel PRO/1000MT QP (82546EB)" rev 0x01: irq
11, a
ddress: 00:04:23:c4:5b:67
uhci0 at pci0 dev 29 function 0 "Intel 6300ESB USB" rev 0x02: irq 3
usb0 at uhci0: USB revision 1.0
uhub at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 5300ESB USB" rev 0x02: irq 11
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
"Intel 6300ESB WDT" rev 0x02 at pci0 dev 29 function 4 not configured
"Intel 6300ESB APIC" rev 0x02 at pci0 dev 29 function 5 not configured
ehci0 at pci0 dev 29 function 7 "Intel 6300ESB USB" rev 0x02: irq 7
usb2 at ehci: USB revision 2.0
uhub2 at usb2
uhub2: Intel EHCI root hub, rev 2.00/1.00, addr 1
uhub2: 4 ports with 4 removable, self powered
ppb3 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x0a
pci4 at ppb3 bus
vga1 at pci4 dev 2 function 0 "ATI Radeon VE QY" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt1000 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
em5 at pci4 dev 3 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00: irq 11,
addr
ess: 00:14:5e:41:7a:d3
ichpcib0 at pci0 dev 31 function 0 "Intel 6300ESB LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 6300ESB SATA" rev 0x02: DMA,
channel 0
configured to compability, channel 1 configured to compability
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <HL-DT-ST, CD-ROM GCR-8240N, 1.06> SCSI0
5/cdrom r
emovable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
wd0 at pciide0 channel 1 drive 0: <Maxtor 6Y080M0>
wd0: 16-sector PIO, LBA, 76324MB, 156312576 sectors
wd1 at pciide0 channel 1 drive 1: <Maxtor 6Y080M0>
wd1: 16-sector PIO, LBA; 76324MB, 156312576 sectors
wd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 5
wd1(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 5
"Intel 6300ESB SMBus" rev 0x02 at pci0 dev 31 function 3 not configured
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask efcd netmask efed ttymask ffef
pctr: user-level cycle counter enabled
dkcsum: wd0 matches BIOS drive 0x80
dkcsum: wd1 matches BIOS drive 0x81
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
WARNING: / was not properly unmounted
Automatic boot in progress: starting file system checks.
/dev/rwd0a: FREE BLK COUNTS(S) WRONG IN SUPERBLK (SALVAGED)
/dev/rwd0a: SUMMARY INFORMATION BAD (SALVAGED)
/dev/rwd0a: BKL(S) MISSING IN BIT MAPS (SALVAGED)
/dev/rwd0a: 89413 files, 483975 used, 35636301 free (589 frags, 4454464
blocks,
0.0% fragmentation)
/dev/rwd0a: MARKING FILE SYSTEM CLEAN
setting tty flags
kbd: keyboard mapping set to sv
pf enabled
net.inet.ip.forwarding: 0 -> 1
net.inet.carp.preempt: 0 -> 1
net.inet.carp.arpbalance: 0 -> 1
net.inet.carp.log: 0 -> 1
kern.maxclusters: 6144 -> 15000
starting network
starting system logger
starting rpc daemons: ntpd.
savecore: no core dump
checking quotas: done
building ps databases: kvm dev.
clearing /tmp
starting pre-securelevel daemons:.
setting kernel security level: kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files
starting network daemons: sendmail inetd sshd.
starting local daemons:.
add net xxx.xxx.xxx.xxx/xx: gateway xxx.xxx.xxx.xxx
add net xxx.xxx.xxx.xxx/xx: gateway xxx.xxx.xxx.xxx
add net xxx.xxx.xxx.xxx/xx: gateway xxx.xxx.xxx.xxx
add net xxx.xxx.xxx.xxx/xx: gateway xxx.xxx.xxx.xxx
standard daemons: cron.
Thu Aug 31 15:30:46 CEST 2006
OpenBSD/i386 (fw0-host.my.domain) (ttyC0)
login:
