see archives on misc@ about month ago or so there was thread about something similar. New SMP HW with a lot of cores doesn't mean automatically better FW or speed.
On Mon, Mar 28, 2011 at 11:27 AM, Peter Hallin <peter.hal...@ldc.lu.se> wrote: > Hello all, > > Last saturday during a service window, we installed a new firewall as a > replacement to one of our oldest firewalls that had been running at a > very high load for a long time. > > The old fw ran 3.9 (i386, SP) on a Pentium4 3.2Ghz and was able to > forward data at about 300 Mbit/s at 100% CPu usage (interrupts). The > NICs in this machine was Intel 1000/Pro cards on the PCI bus. > > We were hoping that the new hardware would be able to achieve a higher > throughput, perhaps near 1 Gbit/s. The result, however was even worse > than the old fw. We can only get about 70-80 Mbit/s at 100% CPU... > > The new system has a Xeon 4-core CPU at 2,5 Ghz and Intel 1000/Pro cards > on the PCI-E bus. It's running 4.8 GENERIC.MP amd64. > > What we noticed when checking systat was a high number of livelocks on > the fw interfaces, em0 and em2: > > IFACE B B B B B B LIVELOCKS B SIZE ALIVE B LWM B HWM B CWM > System B B B B B B B B B B B B 256 B B 50 B B B B 458 > B B B B B B B B B B B B B B B 2k B B 30 B B B B 2142 > lo0 > em0 B B B B B B B B B 313390 B B 2k B B 84 B 256 B B 8 > em1 > em2 B B B B B B B B B 379051 B B 2k B B 11 B B B B B 4 B 256 B B 11 > em3 > em4 B B B B B B B B B B 4962 B B 2k B B 4 B B B B B 4 B 256 B B 4 > > This seems extremely high compared to some of our other fws where the > number of livelocks are a couple of hundreds after months of uptime. > > What could be the reason for this huge drop in performance? I know that > the kernel only uses on CPU core and the clock frequency is a little > lower, but it seems the difference is too high. > > Could it be the PCI-E NICs? The bus speed is much faster than the old > PCI cards, so we hoped they would perform better. > > Could it be the OS? 64 bits instead of 32 and MP kernel instead of SP? > > What we do know is that it's not PF, as we disabled it for a couple of > minutes and still had 100% interrupts. > > Do any of you have the same issues with high interrupts and low > throughput? > > We really don't know where to start... :( > > I'm very grateful for any kind of input regarding this matter. > > Brgds, Peter Hallin, Lund University Sweden > > dmesg: > > OpenBSD 4.8 (GENERIC.MP) #335: Mon Aug 16 09:09:20 MDT 2010 > B B dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > B B B B B B B B RTC BIOS diagnostic error 2 > B B B B B B B B real mem = 2139598848 (2040MB) > B B B B B B B B avail mem = 2068819968 (1972MB) > B B B B B B B B mainbus0 at root > B B B B B B B B bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7fa32000 (60 entries) > B B B B B B B B bios0: vendor Intel Corporation version "S5000.86B.11.00.0096.011320091422" date 01/13/2009 > B B B B B B B B bios0: Intel S5000VSA > B B B B B B B B acpi0 at bios0: rev 2 > B B B B B B B B acpi0: sleep states S0 S1 S4 S5 > B B B B B B B B acpi0: tables DSDT FACP APIC SPCR HPET MCFG SSDT SSDT HEST BERT ERST EINJ > B B B B B B B B acpi0: wakeup devices SLPB(S5) P32_(S5) PS2M(S1) PS2K(S1) UAR1(S5) UAR2(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S5) PCIO(S5) PCIP(S5) PCIQ(S5) > B B B B B B B B acpitimer0 at acpi0: 3579545 Hz, 24 bits > B B B B B B B B acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > B B B B B B B B cpu0 at mainbus0: apid 0 (boot processor) > B B B B B B B B cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.09 MHz > B B B B B B B B cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG > B B B B B B B B cpu0: 6MB 64b/line 16-way L2 cache > B B B B B B B B cpu0: apic clock running at 332MHz > B B B B B B B B cpu1 at mainbus0: apid 2 (application processor) > B B B B B B B B cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz > B B B B B B B B cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG > B B B B B B B B cpu1: 6MB 64b/line 16-way L2 cache > B B B B B B B B cpu2 at mainbus0: apid 1 (application processor) > B B B B B B B B cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz > B B B B B B B B cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG > B B B B B B B B cpu2: 6MB 64b/line 16-way L2 cache > B B B B B B B B cpu3 at mainbus0: apid 3 (application processor) > B B B B B B B B cpu3: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz > B B B B B B B B cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG > B B B B B B B B cpu3: 6MB 64b/line 16-way L2 cache > B B B B B B B B ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins > B B B B B B B B ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins > B B B B B B B B acpihpet0 at acpi0: 14318179 Hz > B B B B B B B B acpiprt0 at acpi0: bus 0 (PCI0) > B B B B B B B B acpiprt1 at acpi0: bus 9 (P32_) > B B B B B B B B acpiprt2 at acpi0: bus 8 (PEX0) > B B B B B B B B acpiprt3 at acpi0: bus -1 (PEX1) > B B B B B B B B acpiprt4 at acpi0: bus -1 (PEX2) > B B B B B B B B acpiprt5 at acpi0: bus -1 (PEX3) > B B B B B B B B acpiprt6 at acpi0: bus 1 (PCIE) > B B B B B B B B acpiprt7 at acpi0: bus 6 (PCIE) > B B B B B B B B acpiprt8 at acpi0: bus 2 (PCIW) > B B B B B B B B acpiprt9 at acpi0: bus 3 (PCIO) > B B B B B B B B acpiprt10 at acpi0: bus 4 (PCIP) > B B B B B B B B acpiprt11 at acpi0: bus 5 (PCIQ) > B B B B B B B B acpicpu0 at acpi0: C2, C1 > B B B B B B B B acpicpu1 at acpi0: C2, C1 > B B B B B B B B acpicpu2 at acpi0: C2, C1 > B B B B B B B B acpicpu3 at acpi0: C2, C1 > B B B B B B B B acpibtn0 at acpi0: SLPB > B B B B B B B B ipmi at mainbus0 not configured > B B B B B B B B pci0 at mainbus0 bus 0 > B B B B B B B B pchb0 at pci0 dev 0 function 0 "Intel 5000V Host" rev 0xb1 > B B B B B B B B ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0xb1 > B B B B B B B B pci1 at ppb0 bus 1 > B B B B B B B B ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 > B B B B B B B B pci2 at ppb1 bus 2 > B B B B B B B B ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 16 (irq 10) > B B B B B B B B pci3 at ppb2 bus 3 > B B B B B B B B em0 at pci3 dev 0 function 0 "Intel PRO/1000 (82576)" rev 0x01: apic 8 int 16 (irq 10), address 00:1b:21:9e:36:cc > B B B B B B B B em1 at pci3 dev 0 function 1 "Intel PRO/1000 (82576)" rev 0x01: apic 8 int 17 (irq 11), address 00:1b:21:9e:36:cd > B B B B B B B B ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 17 (irq 11) > B B B B B B B B pci4 at ppb3 bus 4 > B B B B B B B B em2 at pci4 dev 0 function 0 "Intel PRO/1000 (82576)" rev 0x01: apic 8 int 17 (irq 11), address 00:1b:21:9e:35:ba > B B B B B B B B em3 at pci4 dev 0 function 1 "Intel PRO/1000 (82576)" rev 0x01: apic 8 int 18 (irq 5), address 00:1b:21:9e:35:bb > B B B B B B B B ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01 > B B B B B B B B pci5 at ppb4 bus 5 > B B B B B B B B em4 at pci5 dev 0 function 0 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 18 (irq 5), address 00:15:17:89:eb:fc > B B B B B B B B em5 at pci5 dev 0 function 1 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 19 (irq 11), address 00:15:17:89:eb:fd > B B B B B B B B ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01 > B B B B B B B B pci6 at ppb5 bus 6 > B B B B B B B B ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1 > B B B B B B B B pci7 at ppb6 bus 7 > B B B B B B B B "Intel I/OAT" rev 0xb1 at pci0 dev 8 function 0 not configured > B B B B B B B B pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0xb1 > B B B B B B B B pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0xb1 > B B B B B B B B pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0xb1 > B B B B B B B B pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0xb1 > B B B B B B B B pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0xb1 > B B B B B B B B pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0xb1 > B B B B B B B B pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0xb1 > B B B B B B B B ppb7 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09: apic 8 int 16 (irq 10) > B B B B B B B B pci8 at ppb7 bus 8 > B B B B B B B B uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) > B B B B B B B B uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 22 (irq 5) > B B B B B B B B uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) > B B B B B B B B uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09: apic 8 int 22 (irq 5) > B B B B B B B B ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) > B B B B B B B B usb0 at ehci0: USB revision 2.0 > B B B B B B B B uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > B B B B B B B B ppb8 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9 > B B B B B B B B pci9 at ppb8 bus 9 > B B B B B B B B vga1 at pci9 dev 12 function 0 "ATI ES1000" rev 0x02 > B B B B B B B B wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > B B B B B B B B wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > B B B B B B B B radeondrm0 at vga1: apic 8 int 17 (irq 11) > B B B B B B B B drm0 at radeondrm0 > B B B B B B B B pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09 > B B B B B B B B pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility > B B B B B B B B wd0 at pciide0 channel 0 drive 0: <WDC WD2500AAJB-00J3A0> > B B B B B B B B wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors > B B B B B B B B wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 > B B B B B B B B pciide0: channel 1 disabled (no drives) > B B B B B B B B pciide1 at pci0 dev 31 function 2 "Intel 6321ESB SATA" rev 0x09: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI > B B B B B B B B pciide1: using apic 8 int 20 (irq 10) for native-PCI interrupt > B B B B B B B B atapiscsi0 at pciide1 channel 0 drive 0 > B B B B B B B B scsibus0 at atapiscsi0: 2 targets > B B B B B B B B cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, CDDVDW SH-S223F, SB02> ATAPI 5/cdrom removable > B B B B B B B B cd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5 > B B B B B B B B ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int 20 (irq 10) > B B B B B B B B iic0 at ichiic0 > B B B B B B B B usb1 at uhci0: USB revision 1.0 > B B B B B B B B uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > B B B B B B B B usb2 at uhci1: USB revision 1.0 > B B B B B B B B uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > B B B B B B B B usb3 at uhci2: USB revision 1.0 > B B B B B B B B uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > B B B B B B B B usb4 at uhci3: USB revision 1.0 > B B B B B B B B uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > B B B B B B B B isa0 at pcib0 > B B B B B B B B isadma0 at isa0 > B B B B B B B B com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > B B B B B B B B com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > B B B B B B B B pckbc0 at isa0 port 0x60/5 > B B B B B B B B pckbd0 at pckbc0 (kbd slot) > B B B B B B B B pckbc0: using irq 1 for kbd slot > B B B B B B B B wskbd0 at pckbd0: console keyboard, using wsdisplay0 > B B B B B B B B pmsi0 at pckbc0 (aux slot) > B B B B B B B B pckbc0: using irq 12 for aux slot > B B B B B B B B wsmouse0 at pmsi0 mux 0 > B B B B B B B B pcppi0 at isa0 port 0x61 > B B B B B B B B spkr0 at pcppi0 > B B B B B B B B mtrr: Pentium Pro MTRR support > B B B B B B B B softraid0 at root > B B B B B B B B root on wd0a swap on wd0b dump on wd0b > > B B B B B B B B END dmesg.
