One of my carp masters crashed last night. Fortunately, the slave
immediately picked up and nobody was the wiser.
Hardware: Dell Poweredge 1850 (2.8Ghz Xeon, 1G RAM, 1x 36G U360 15K RPM
SCSI, 2x onboard em, 1x dual intel em card, 1x quad intel em card,
attached 16x usb serial console device)
Not too sure what the machine was doing network-wise at the time, but
chances are it was pushing quite a bit of traffic (maybe 10Mb/s or more
-- reportedly, this was right about the time 12 machines all tried to
read the same very large file on a NFS server that this firewall
protects). Two clues I have are that another pair of carp'd firewalls
that were helping push all this traffic flopped back and forth as
master/slave and the following last log message from the firewall in
this pair that failed:
Dec 27 18:07:47 xxx-fw-1 /bsd: WARNING: mclpool limit reached; increase
kern.maxclusters
Below is a trace, limited ps (machine got booted prior to me finishing
getting what I needed) and a dmesg. I'm not sure if this condition is
repeatable or not but will likely know later today after I do more
testing.
Any insight would be much appreciated!
-jon
trace:
kernel: page fualt trap, code=0
stopped at ether_input+0x3d: testb $0x1,0(%ecx)
trace:
ether_input(d18a1830,0,e9806400,be,bf) at ether_input+0x3d
ether_input_mbuf(d18a1830,e9806400,e9806400,d1a29800) at
ether_input_mbuf+0x23
em_process_receive_interrupts(d18a1800,fffffffc,e9037cbc,d0346b55,b0) at
em_process_receive_interrrupts+0x190
em_inter(d18a1800) at em_intr+0x93
Xintr_ioapic5() at Xintr_ioapic5+0x74
--- interrupt ---
bcopy(d18a3030,d7002700,d1967b50,d7ade220,30) at bcopy+0x20
ip_output(d7031400,0,d05d37e4,1,0,0,0,0) at ip_output+0x87a
ip_forward(d7031400,0,0,50,e9037f00) at ip_forward+0x159
ipv4_input(d7031400,30,3,0) at ipv4_input+0x25a
ipintr(d18a0058,10,e9030010,e9030010,e9036000) at ipintr+0x67
Bad frame pointer: 0xe9037f20
ps
8123 1 12702 0 3 0x2000084 select mrouted
17362 1 17362 0 3 0x2000084 kqread ifstated
16250 3631 3631 74 3 0x2000184 bpf pflogd
3631 1 3631 0 3 0x2000084 netio pflogd
poll syslogd
netio syslogd
crypt_wa crypto
aidoned aidoned
syncer update
cleaner cleaner
reaper repaper
pgdaemon pagedaemon
pftm pfpurge
usbevt usb3
usbevt usb2
usbevt usb1
usbtsk usbtsk
usbevt usb0
kmalloc kmthread
wait init
scheduler swapper
dmesg:
OpenBSD 3.8-current (GENERIC.MP) #353: Tue Oct 18 05:16:18 MDT 2005
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
real mem = 1073065984 (1047916K)
avail mem = 972525568 (949732K)
using 4278 buffers containing 53755904 bytes (52496K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 01/19/05, BIOS32 rev.
0 @ 0xffe90
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb460/256 (14 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC" rev
0x00)
pcibios0: PCI bus #10 is the last bus
bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x4000 0xec000/0x4000!
ipmi0 at mainbus0: version 1.5 interface kcs ibase 0xca8/8 spacing 4 irq
-1
mainbus0: Intel MP Specification (Version 1.4) (DELL PE 016C )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 199 MHz
mainbus0: bus 0 is type PCI
mainbus0: bus 1 is type PCI
mainbus0: bus 2 is type PCI
mainbus0: bus 3 is type PCI
mainbus0: bus 4 is type PCI
mainbus0: bus 5 is type PCI
mainbus0: bus 6 is type PCI
mainbus0: bus 7 is type PCI
mainbus0: bus 8 is type PCI
mainbus0: bus 9 is type PCI
mainbus0: bus 10 is type PCI
mainbus0: bus 11 is type ISA
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apic 2
ioapic1 at mainbus0: apid 3 pa 0xfec80000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apic 3
ioapic2 at mainbus0: apid 4 pa 0xfec83000, version 20, 24 pins
ioapic2: misconfigured as apic 0, remapped to apic 4
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7710 SMCH" rev 0x09
ppb0 at pci0 dev 2 function 0 "Intel E7710 MCH PCIE" rev 0x09
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci2 at ppb1 bus 2
ppb2 at pci2 dev 11 function 0 "IBM PCIX-PCIX" rev 0x02
pci3 at ppb2 bus 3
em0 at pci3 dev 4 function 0 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
apic 3 int 5 (irq 3), address 00:04:23:ba:9a:ac
em1 at pci3 dev 4 function 1 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
apic 3 int 6 (irq 7), address 00:04:23:ba:9a:ad
em2 at pci3 dev 6 function 0 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
apic 3 int 7 (irq 10), address 00:04:23:ba:9a:ae
em3 at pci3 dev 6 function 1 "Intel PRO/1000MT QP (82546EB)" rev 0x01:
apic 3 int 4 (irq 11), address 00:04:23:ba:9a:af
ppb3 at pci1 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci4 at ppb3 bus 4
mpt0 at pci4 dev 5 function 0 "Symbios Logic 53c1030" rev 0x08: apic
3 int 2 (irq 7)
mpt0: sending FW Upload request to IOC (size: 36, img size: 40048)
mpt0: IM support: 0
scsibus0 at mpt0: 16 targets
sd0 at scsibus0 targ 0 lun 0: <MAXTOR, ATLAS15K2_36SCA, JNZM> SCSI3
0/direct fixed
sd0: 34732MB, 48242 cyl, 2 head, 737 sec, 512 bytes/sec, 71132959 sec
total
sd1 at scsibus0 targ 6 lun 0: <, , > SCSI0 0/direct fixed
d1(mpt0:6:0): could not get size
sd1: drive offline
mpt0: target 0 Synchronous at 160MHz width 16bit offset 127 QAS 0 DT
1 IU 1
mpt0: target 6 Asynchronous at 0MHz width 8bit offset 0 QAS 0 DT 0 IU 0
em4 at pci4 dev 12 function 0 "Intel PRO/1000MT (82546GB)" rev 0x03:
apic 3 int 7 (irq 10), address 00:04:23:bd:1e:6a
em5 at pci4 dev 12 function 1 "Intel PRO/1000MT (82546GB)" rev 0x03:
apic 3 int 4 (irq 11), address 00:04:23:bd:1e:6b
ppb4 at pci0 dev 4 function 0 "Intel E7710 MCH PCIE" rev 0x09
pci5 at ppb4 bus 5
ppb5 at pci0 dev 5 function 0 "Intel E7710 MCH PCIE" rev 0x09
pci6 at ppb5 bus 6
ppb6 at pci6 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci7 at ppb6 bus 7
em6 at pci7 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: apic
4 int 0 (irq 11), address 00:14:22:11:26:04
ppb7 at pci6 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci8 at ppb7 bus 8
em7 at pci8 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: apic
4 int 1 (irq 3), address 00:14:22:11:26:05
ppb8 at pci0 dev 6 function 0 "Intel E7710 MCH PCIE" rev 0x09
pci9 at ppb8 bus 9
uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 16 (irq 11)
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 19 (irq 10)
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 18 (irq 7)
usb2 at uhci2: USB revision 1.0
uhub2 at usb2
uhub2: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB" rev 0x02: apic
2 int 23 (irq 5)
usb3 at ehci0: USB revision 2.0
uhub3 at usb3
uhub3: Intel EHCI root hub, rev 2.00/1.00, addr 1
uhub3: 6 ports with 6 removable, self powered
ppb9 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
pci10 at ppb9 bus 10
vga1 at pci10 dev 13 function 0 "ATI Radeon VE QY" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <TEAC, CD-224E, K.9A> SCSI0 5/cdrom
removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask 0 netmask 0 ttymask 0
pctr: user-level cycle counter enabled
uhub4 at uhub3 port 3
uhub4: Dell product 0xa001, rev 2.00/0.00, addr 2
uhub4: 2 ports with 2 removable, self powered, multiple transaction
translators
uhub5 at uhub2 port 1
uhub5: Genesys Logic GL650 Hub, rev 1.01/0.11, addr 2
uhub5: 7 ports with 7 removable, self powered
uftdi0 at uhub5 port 1 configuration 1 interface 0
uftdi0: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 3
ucom0 at uftdi0 portno 1
uftdi1 at uhub5 port 1 configuration 1 interface 1
uftdi1: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 3
ucom1 at uftdi1 portno 2
uftdi2 at uhub5 port 2 configuration 1 interface 0
uftdi2: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 4
ucom2 at uftdi2 portno 1
uftdi3 at uhub5 port 2 configuration 1 interface 1
uftdi3: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 4
ucom3 at uftdi3 portno 2
uftdi4 at uhub5 port 3 configuration 1 interface 0
uftdi4: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 5
ucom4 at uftdi4 portno 1
uftdi5 at uhub5 port 3 configuration 1 interface 1
uftdi5: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 5
ucom5 at uftdi5 portno 2
uftdi6 at uhub5 port 4 configuration 1 interface 0
uftdi6: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 6
ucom6 at uftdi6 portno 1
uftdi7 at uhub5 port 4 configuration 1 interface 1
uftdi7: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 6
ucom7 at uftdi7 portno 2
uftdi8 at uhub5 port 5 configuration 1 interface 0
uftdi8: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 7
ucom8 at uftdi8 portno 1
uftdi9 at uhub5 port 5 configuration 1 interface 1
uftdi9: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 7
ucom9 at uftdi9 portno 2
uftdi10 at uhub5 port 6 configuration 1 interface 0
uftdi10: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 8
ucom10 at uftdi10 portno 1
uftdi11 at uhub5 port 6 configuration 1 interface 1
uftdi11: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 8
ucom11 at uftdi11 portno 2
uhub6 at uhub5 port 7
uhub6: Genesys Logic GL650 Hub, rev 1.01/0.12, addr 9
uhub6: 4 ports with 4 removable, self powered
uftdi12 at uhub6 port 1 configuration 1 interface 0
uftdi12: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 10
ucom12 at uftdi12 portno 1
uftdi13 at uhub6 port 1 configuration 1 interface 1
uftdi13: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 10
ucom13 at uftdi13 portno 2
uftdi14 at uhub6 port 2 configuration 1 interface 0
uftdi14: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 11
ucom14 at uftdi14 portno 1
uftdi15 at uhub6 port 2 configuration 1 interface 1
uftdi15: FTDI USB FAST SERIAL ADAPTER, rev 2.00/5.00, addr 11
ucom15 at uftdi15 portno 2
dkcsum: sd0 matches BIOS drive 0x80
sd1(mpt0:6:0): Check Condition (error 0x70) on opcode 0x25
SENSE KEY: Illegal Request
ASC/ASCQ: No Additional Sense Information
sd1(mpt0:6:0): could not get size
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
WARNING: / was not properly unmounted
