We have two OpenBSD firewalls running in CARP redundant mode, one
active, one standby. The problem we've been seeing for a while
appears to be packet loss at our firewall once we reach or surpass
around 100Mbps / 12k pps. I've seen this show up on both 3.9 stock
and the download of 4.0. It is replicable on both boxes.
Here's what shows up on the interfaces once we hit the above
threshold. em2 is the outside interface, em4 is the inside interface.
Note the number of Ierrs; that is what increments on either or both
interfaces once we hit the above traffic/pps level. We occasionally
will get Oerrs as well, but not as common.
$ netstat -idq
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls Drop
<snip>
em2 1500 <Link> 00:04:23:c2:4c:2a 3191117038 4335
2758336584 0 0 0
<snip>
em4 1500 <Link> 00:04:23:c2:45:5e 2941667842 2536
3337201967 0 0 0
Also, when the above happens, if you run 'netstat -I <interface> -w
1', you will see the packet rate drop from 12-13k pps to down to 3-5k
pps
We purchased machines with PCI-X riser card and Intel PRO-MT dual port
Gigabit cards, so I was hoping that previous posts referencing 250k+
pps would have been an indicator of performance, and that we should at
least be able to get 50k+ pps or so without an issue.
We are running PF and doing NAT. Filter ruleset is about 120 lines,
NAT ruleset about 70.
The problem has duplicated itself on both firewalls (it follows
regardless which one is the master. The two GigE interfaces on the
firewalls are connected to a Cisco 2970, and none of the Cisco switch
interfaces are showing any kinds of errors at all. The patch cables
are all brand new Cat6. Duplex/speed is set to auto everywhere,
negotiating properly to 1000/full.
This is causing headaches because we have video conferencing that uses
UDP going on during peak traffic production, and dropping of packets
is causing video performance that is not acceptable to our users.
We've verified patch cables from the firewalls to the switch, as well
as the routers upstream and downstream (no errors on any of those
interfaces).
This is right now (currently doing about 95Mb, 11k pps), just saw
about 500 Ierrs:
$ netstat -m
787 mbufs in use:
782 mbufs allocated to data
2 mbufs allocated to packet headers
3 mbufs allocated to socket names and addresses
782/1032/6144 mbuf clusters in use (current/peak/max)
2336 Kbytes allocated to network (75% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
Here is a dmesg from both our 3.9 and 4.0 versions. Hardware in the
two boxes is identical:
3.9
----
OpenBSD 3.9 (GENERIC.MP) #598: Thu Mar 2 02:37:06 MST 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
real mem = 2146807808 (2096492K)
avail mem = 1952759808 (1906992K)
using 4278 buffers containing 107442176 bytes (104924K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 01/09/06, BIOS32 rev. 0 @ 0xffe90
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb4b0/320 (18 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC" rev 0x00)
pcibios0: PCI bus #11 is the last bus
bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x2200 0xec000/0x4000!
ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca8/8 spacing 4
mainbus0: Intel MP Specification (Version 1.4) (DELL PE 016D )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 199 MHz
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,CNXT-ID
mainbus0: bus 0 is type PCI
mainbus0: bus 1 is type PCI
mainbus0: bus 2 is type PCI
mainbus0: bus 3 is type PCI
mainbus0: bus 4 is type PCI
mainbus0: bus 5 is type PCI
mainbus0: bus 6 is type PCI
mainbus0: bus 7 is type PCI
mainbus0: bus 8 is type PCI
mainbus0: bus 9 is type PCI
mainbus0: bus 10 is type PCI
mainbus0: bus 11 is type PCI
mainbus0: bus 12 is type ISA
ioapic0 at mainbus0: apid 7 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apic 7
ioapic1 at mainbus0: apid 8 pa 0xfec80000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apic 8
ioapic2 at mainbus0: apid 9 pa 0xfec83000, version 20, 24 pins
ioapic2: misconfigured as apic 0, remapped to apic 9
ioapic3 at mainbus0: apid 10 pa 0xfec84000, version 20, 24 pins
ioapic3: misconfigured as apic 0, remapped to apic 10
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x09
ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x09
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel IOP331 Channel 0" rev 0x06
pci2 at ppb1 bus 2
ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: apic 8 int
14 (irq 11) Dell 16d 32b
ami0: FW 521X, BIOS vH430, 256MB RAM
ami0: 2 channels, 0 FC loops, 1 logical drives
scsibus0 at ami0: 40 targets
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 69880MB, 69880 cyl, 64 head, 32 sec, 512 bytes/sec, 143114240 sec total
scsibus1 at ami0: 16 targets
safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x6 SCSI BP, 1.0> SCSI2
3/processor fixed
scsibus2 at ami0: 16 targets
ppb2 at pci1 dev 0 function 2 "Intel IOP331 Channel 1" rev 0x06
pci3 at ppb2 bus 3
ppb3 at pci0 dev 4 function 0 "Intel MCH PCIE" rev 0x09
pci4 at ppb3 bus 4
ppb4 at pci0 dev 5 function 0 "Intel MCH PCIE" rev 0x09
pci5 at ppb4 bus 5
ppb5 at pci5 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci6 at ppb5 bus 6
em0 at pci6 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
apic 9 int 0 (irq 10), address 00:14:22:1c:c9:56
ppb6 at pci5 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci7 at ppb6 bus 7
em1 at pci7 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
apic 9 int 1 (irq 7), address 00:14:22:1c:c9:57
ppb7 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x09
pci8 at ppb7 bus 8
ppb8 at pci8 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci9 at ppb8 bus 9
em2 at pci9 dev 4 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 10 (irq 11), address 00:04:23:c2:4b:2a
em3 at pci9 dev 4 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 11 (irq 5), address 00:04:23:c2:4b:2b
ppb9 at pci8 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci10 at ppb9 bus 10
em4 at pci10 dev 2 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 0 (irq 10), address 00:04:23:c2:4a:64
em5 at pci10 dev 2 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 1 (irq 7), address 00:04:23:c2:4a:65
ppb10 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
pci11 at ppb10 bus 11
vga1 at pci11 dev 13 function 0 "ATI Radeon VE QY" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
DMA, channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <TEAC, CD-ROM CD-224E-N, 3.AB> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom0: console
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask 0 netmask 0 ttymask 0
pctr: user-level cycle counter enabled
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
4.0
-----
OpenBSD 4.0 (GENERIC.MP) #933: Fri Sep 1 12:06:05 MDT 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16
real mem = 2146795520 (2096480K)
avail mem = 1950171136 (1904464K)
using 4256 buffers containing 107442176 bytes (104924K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 09/22/05, BIOS32 rev. 0 @
0xffe90, SMBIOS rev. 2.3 @ 0xf9920 (87 entries)
bios0: Dell Computer Corporation PowerEdge 2850
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfb4b0/320 (18 entries)
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82801EB/ER LPC" rev 0x00)
pcibios0: PCI bus #11 is the last bus
bios0: ROM list: 0xc0000/0xb000! 0xcb000/0x2200 0xec000/0x4000!
ipmi0 at mainbus0: version 1.5 interface KCS iobase 0xca8/8 spacing 4
mainbus0: Intel MP Specification (Version 1.4) (DELL PE 016D )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 199 MHz
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,CX16
mainbus0: bus 0 is type PCI
mainbus0: bus 1 is type PCI
mainbus0: bus 2 is type PCI
mainbus0: bus 3 is type PCI
mainbus0: bus 4 is type PCI
mainbus0: bus 5 is type PCI
mainbus0: bus 6 is type PCI
mainbus0: bus 7 is type PCI
mainbus0: bus 8 is type PCI
mainbus0: bus 9 is type PCI
mainbus0: bus 10 is type PCI
mainbus0: bus 11 is type PCI
mainbus0: bus 12 is type ISA
ioapic0 at mainbus0: apid 7 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 7
ioapic1 at mainbus0: apid 8 pa 0xfec80000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apid 8
ioapic2 at mainbus0: apid 9 pa 0xfec83000, version 20, 24 pins
ioapic2: misconfigured as apic 0, remapped to apid 9
ioapic3 at mainbus0: apid 10 pa 0xfec84000, version 20, 24 pins
ioapic3: misconfigured as apic 0, remapped to apid 10
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7520 MCH" rev 0x09
ppb0 at pci0 dev 2 function 0 "Intel MCH PCIE" rev 0x09
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel IOP332 PCIE-PCIX" rev 0x06
pci2 at ppb1 bus 2
ami0 at pci2 dev 14 function 0 "Dell PERC 4e/Di" rev 0x06: apic 8 int
14 (irq 11)
ami0: Dell 16d, 32b, FW 521S, BIOS vH430, 256MB RAM
ami0: 2 channels, 0 FC loops, 1 logical drives
scsibus0 at ami0: 40 targets
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 69880MB, 69880 cyl, 64 head, 32 sec, 512 bytes/sec, 143114240 sec total
scsibus1 at ami0: 16 targets
safte0 at scsibus1 targ 6 lun 0: <PE/PV, 1x6 SCSI BP, 1.0> SCSI2
3/processor fixed
scsibus2 at ami0: 16 targets
ppb2 at pci1 dev 0 function 2 "Intel IOP332 PCIE-PCIX" rev 0x06
pci3 at ppb2 bus 3
ppb3 at pci0 dev 4 function 0 "Intel MCH PCIE" rev 0x09
pci4 at ppb3 bus 4
ppb4 at pci0 dev 5 function 0 "Intel MCH PCIE" rev 0x09
pci5 at ppb4 bus 5
ppb5 at pci5 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci6 at ppb5 bus 6
em0 at pci6 dev 7 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
apic 9 int 0 (irq 10), address 00:14:22:1d:2f:1c
ppb6 at pci5 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci7 at ppb6 bus 7
em1 at pci7 dev 8 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05:
apic 9 int 1 (irq 7), address 00:14:22:1d:2f:1d
ppb7 at pci0 dev 6 function 0 "Intel MCH PCIE" rev 0x09
pci8 at ppb7 bus 8
ppb8 at pci8 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci9 at ppb8 bus 9
em2 at pci9 dev 4 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 10 (irq 11), address 00:04:23:c2:4c:2a
em3 at pci9 dev 4 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 11 (irq 5), address 00:04:23:c2:4c:2b
ppb9 at pci8 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci10 at ppb9 bus 10
em4 at pci10 dev 3 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 5 (irq 7), address 00:04:23:c2:45:5e
em5 at pci10 dev 3 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01:
apic 10 int 6 (irq 11), address 00:04:23:c2:45:5f
ppb10 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
pci11 at ppb10 bus 11
vga1 at pci11 dev 13 function 0 "ATI Radeon VE QY" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02:
DMA, channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <TEAC, CD-ROM CD-224E-N, 3.AB> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom0: console
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask 0 netmask 0 ttymask 0
pctr: user-level cycle counter enabled
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
