Hi David Thank you for the quick reply!
The stack trace from the console at the time of the crash: Starting stack trace panic(d07a8c58,0,de1b4b28,0,d8a282b8) at panic+0x65 panic(d071ad67,6,0,d031baf5,de1b4b20) at panic+0x65 trap() at trap+0x119 --- trap (number 6) --- pfsync_state_import(d899e83a,2,2d0,de1b4bc8) at pfsync_state_import+0x75 pfsync_in_ins(de1b4c00,d8a3d400,2c,3) at pfsync_in_ins+0xe3 pfsync_input(d8a3d400,14,0,0,d2a8d030) at pfsync_input+0x148 ipv4_input(d8a3d400,d2a82440,f0279,f0275) at ipv4input+0x498 ipintr(58,d8310010,de1b0010,d0350010,f0275) at ipintr+0x64 bad fram pointer: 0xde1b4c88 End of stack trace And here is the output from cvs status if_pfsync.? # cvs status if_pfsync.? =================================================================== File: if_pfsync.c Status: Up-to-date Working revision: 1.127 Repository revision: 1.127 /cvs/src/sys/net/if_pfsync.c,v Sticky Tag: OPENBSD_4_6 (branch: 1.127.4) Sticky Date: (none) Sticky Options: (none) =================================================================== File: if_pfsync.h Status: Up-to-date Working revision: 1.38 Repository revision: 1.38 /cvs/src/sys/net/if_pfsync.h,v Sticky Tag: OPENBSD_4_6 (branch: 1.38.4) Sticky Date: (none) Sticky Options: (none) Please let me know if there is anything else that I can do to help, Best regards Anders -----Original Message----- From: David Gwynne [mailto:l...@animata.net] Sent: Wednesday, November 18, 2009 01:04 To: Anders Pettersson Cc: misc@openbsd.org Subject: Re: OpenBSD 4.6 pfsync kernel panic hi anders, could you get me a full trace from ddb when the fault occurs? id also like the output of 'cvs info if_pfsync.?' in src/sys/net in the tree you built this kernel from? cheers, dlg On 17/11/2009, at 11:07 PM, Anders Pettersson wrote: > Hi > > We get kernel panics when we reboot either one of our two OpenBSD 4.6 servers > running pf. It seems that the kernel panic always happens at the point where > the pf sync state import happens. Sometimes we can reboot the servers, one at > the time, a number of times in a row without any problems. We have tried to > understand why this occurs but to no avail, is there anyone who could advise > us what to do to try and resolve this? > > The error message say: > > "fatal page fault (6) in supervisor mode > trap type 6 code 0 eip d031baf5 cs 50 eflags 10297 cr2 2c4 cpl 40 > panic: trap type 6, code=0, pc=d031baf5 > . > . > . > --- trap (number 6) --- > pfsync_state_import(d899e83a,2,2d0,de1b4bc8) at pfsync_state_import+0x75" > > We have two identical servers running OpenBSD 4.6 and pf, they are build on > the Supermicro X7SBT motherboard: > http://www.supermicro.com/products/motherboard/Xeon3000/X48/X7SBT.cfm > They have a totalt of six nic; two internal Intel PRO/1000MT (82573E) and > (82573L) and four Intel PRO/1000 QP (82571EB). > > The pfsync interface uses em4 (that is the first of the two internal network > cards - Intel PRO/1000MT (82573E)); > > pfsync0: flags=41<UP,RUNNING> mtu 1500 > priority: 0 > pfsync: syncdev: em4 maxupd: 128 defer: off > groups: carp pfsync > > And we have a simple rule > pass quick on { em4 } proto pfsync > > "ps -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -O paddr" gives; > PID PADDR TT STAT TIME COMMAND > 12176 d89f216c p0 Is+ 0:00.00 (ksh) > 28044 d8ae82c4 C0- R/0 0:01.00 (snortsam) > 18169 d8b9e2c0 C0 Is+ 0:00.00 (ksh) > 3045 d8ae8834 C1 Is+ 0:10.00 (getty) > 10861 d8ae8aec C2 Is+ 0:09.00 (getty) > 3111 d8ae8c48 C3 Is+ 0:09.00 (getty) > 5674 d8ae8da4 C5 Is+ 0:09.00 (getty) > > I have attached the dmesg output from one of the machines at the end of this > email, > > Best regards > > Anders > ------------------------------------ > Mainloop > Anders Pettersson > and...@mainloop.se > Stora Nygatan 5, 2tr > 111 27 Stockholm > Sweden > mobile: +46 (70) 634 5818 > ------------------------------------ > > OpenBSD 4.6-stable (GENERIC.MP) #0: Fri Nov 6 10:18:43 CET 2009 > r...@puffy46.intranet.mainloop.net:/usr/src/sys/arch/i386/compile/GENERIC > .MP > cpu0: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41 > GHz > cpu0: > FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS > H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16, > xTPR > real mem = 2145402880 (2046MB) > avail mem = 2065686528 (1969MB) > mainbus0 at root > bios0 at mainbus0: AT/286+ BIOS, date 12/19/08, BIOS32 rev. 0 @ 0xfdbc0, > SMBIOS rev. 2.5 @ 0x7fedf000 (34 entries) > bios0: vendor Phoenix Technologies LTD version "1.2a" date 12/19/2008 > bios0: Supermicro X7SBT > acpi0 at bios0: rev 2 > acpi0: tables DSDT FACP _MAR MCFG APIC BOOT SPCR ERST HEST BERT EINJ SLIC SSDT > SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT > acpi0: wakeup devices PEG_(S5) PEX_(S5) LAN_(S5) USB4(S5) USB5(S5) USB7(S5) > ESB2(S5) EXP1(S5) EXP5(S5) EXP6(S5) USB1(S5) USB2(S5) USB3(S5) USB6(S5) > ESB1(S5) PCIB(S5) KBC0(S1) MSE0(S1) COM1(S5) COM2(S5) PWRB(S3) > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: apic clock running at 266MHz > cpu1 at mainbus0: apid 1 (application processor) > cpu1: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41 > GHz > cpu1: > FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS > H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16, > xTPR > cpu2 at mainbus0: apid 2 (application processor) > cpu2: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41 > GHz > cpu2: > FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS > H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16, > xTPR > cpu3 at mainbus0: apid 3 (application processor) > cpu3: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41 > GHz > cpu3: > FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS > H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16, > xTPR > ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus 1 (PEG_) > acpiprt2 at acpi0: bus -1 (PEX_) > acpiprt3 at acpi0: bus 5 (EXP1) > acpiprt4 at acpi0: bus 13 (EXP5) > acpiprt5 at acpi0: bus 15 (EXP6) > acpiprt6 at acpi0: bus 17 (PCIB) > acpicpu0 at acpi0: C3, PSS > acpicpu1 at acpi0: C3, PSS > acpicpu2 at acpi0: C3, PSS > acpicpu3 at acpi0: C3, PSS > acpibtn0 at acpi0: PWRB > acpivideo0 at acpi0: IGD0 > bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x1000 0xca000/0x1000 > ipmi at mainbus0 not configured > cpu0: Enhanced SpeedStep 2401 MHz: speeds: 2400, 1600 MHz > pci0 at mainbus0 bus 0: configuration mode 1 (bios) > pchb0 at pci0 dev 0 function 0 "Intel 82X38 Host" rev 0x01 > ppb0 at pci0 dev 1 function 0 "Intel 82X38 PCIE" rev 0x01: apic 4 int 16 (irq > 5) > pci1 at ppb0 bus 1 > ppb1 at pci1 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e > pci2 at ppb1 bus 2 > ppb2 at pci2 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e > pci3 at ppb2 bus 3 > em0 at pci3 dev 0 function 0 "Intel PRO/1000 QP (82571EB)" rev 0x06: apic 4 > int 19 (irq 11), address 00:15:17:a7:97:09 > em1 at pci3 dev 0 function 1 "Intel PRO/1000 QP (82571EB)" rev 0x06: apic 4 > int 18 (irq 11), address 00:15:17:a7:97:08 > ppb3 at pci2 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e > pci4 at ppb3 bus 4 > em2 at pci4 dev 0 function 0 "Intel PRO/1000 QP (82571EB)" rev 0x06: apic 4 > int 17 (irq 10), address 00:15:17:a7:97:0b > em3 at pci4 dev 0 function 1 "Intel PRO/1000 QP (82571EB)" rev 0x06: apic 4 > int 16 (irq 5), address 00:15:17:a7:97:0a > uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 4 int 16 > (irq 5) > uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 4 int 17 > (irq 10) > uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 4 int 18 > (irq 11) > ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 4 int 18 > (irq 11) > usb0 at ehci0: USB revision 2.0 > uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > ppb4 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 4 int 16 > (irq 5) > pci5 at ppb4 bus 5 > ppb5 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 4 int 16 > (irq 5) > pci6 at ppb5 bus 13 > em4 at pci6 dev 0 function 0 "Intel PRO/1000MT (82573E)" rev 0x03: apic 4 int > 16 (irq 5), address 00:30:48:9d:03:d8 > ppb6 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: apic 4 int 17 > (irq 10) > pci7 at ppb6 bus 15 > em5 at pci7 dev 0 function 0 "Intel PRO/1000MT (82573L)" rev 0x00: apic 4 int > 17 (irq 10), address 00:30:48:9d:03:d9 > uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 4 int 23 > (irq 7) > uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 4 int 22 > (irq 10) > uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 4 int 18 > (irq 11) > ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 4 int 23 > (irq 7) > usb1 at ehci1: USB revision 2.0 > uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > ppb7 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92 > pci8 at ppb7 bus 17 > vga1 at pci8 dev 1 function 0 vendor "XGI Technology", unknown product 0x0021 > rev 0x00 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > ichpcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02: PM disabled > pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA, channel 0 > configured to native-PCI, channel 1 configured to native-PCI > pciide0: using apic 4 int 17 (irq 10) for native-PCI interrupt > wd0 at pciide0 channel 0 drive 1: <ST3250310AS> > wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors > wd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 5 > ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 4 int 17 > (irq 10) > iic0 at ichiic0 > lm1 at iic0 addr 0x2d: W83627DHG > spdmem0 at iic0 addr 0x50: 1GB DDR3 SDRAM PC3-10600 > spdmem1 at iic0 addr 0x52: 1GB DDR3 SDRAM PC3-10600 > pciide1 at pci0 dev 31 function 5 "Intel 82801I SATA" rev 0x02: DMA, channel 0 > wired to native-PCI, channel 1 wired to native-PCI > pciide1: using apic 4 int 18 (irq 11) for native-PCI interrupt > "Intel 82801I Thermal" rev 0x02 at pci0 dev 31 function 6 not configured > usb2 at uhci0: USB revision 1.0 > uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb3 at uhci1: USB revision 1.0 > uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb4 at uhci2: USB revision 1.0 > uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb5 at uhci3: USB revision 1.0 > uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb6 at uhci4: USB revision 1.0 > uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb7 at uhci5: USB revision 1.0 > uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > isa0 at ichpcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 > pckbd0 at pckbc0 (kbd slot) > pckbc0: using irq 1 for kbd slot > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pcppi0 at isa0 port 0x61 > midi0 at pcppi0: <PC speaker> > spkr0 at pcppi0 > lm0 at isa0 port 0x290/8: W83627DHG > lm1 detached > npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 > mtrr: Pentium Pro MTRR support > uhidev0 at uhub0 port 4 configuration 1 interface 0 "Peppercon AG Multidevice" > rev 2.00/0.01 addr 2 > uhidev0: iclass 3/0 > ums0 at uhidev0 > ums0: X report 0x0022 not supported > uhidev1 at uhub0 port 4 configuration 1 interface 1 "Peppercon AG Multidevice" > rev 2.00/0.01 addr 2 > uhidev1: iclass 3/1 > ukbd0 at uhidev1: 8 modifier keys, 6 key codes > wskbd1 at ukbd0 mux 1 > wskbd1: connecting to wsdisplay0 > uhidev2 at uhub5 port 2 configuration 1 interface 0 "Avocent Dell 03R874" rev > 1.10/1.00 addr 2 > uhidev2: iclass 3/1 > ukbd1 at uhidev2: 8 modifier keys, 6 key codes, country code 33 > wskbd2 at ukbd1 mux 1 > wskbd2: connecting to wsdisplay0 > uhidev3 at uhub5 port 2 configuration 1 interface 1 "Avocent Dell 03R874" rev > 1.10/1.00 addr 2 > uhidev3: iclass 3/1, 3 report ids > ums1 at uhidev3 reportid 1: 5 buttons, Z dir > wsmouse0 at ums1 mux 0 > uhid0 at uhidev3 reportid 2: input=2, output=0, feature=0 > uhid1 at uhidev3 reportid 3: input=1, output=0, feature=0 > softraid0 at root > root on wd0a swap on wd0b dump on wd0b > WARNING: / was not properly unmounted
