pf and includes
Hello, I have some issues with pf.conf and includes that perhaps someone could shed some light on. Where I work, we use bridging firewalls with multiple tagged vlans passing the bridges, and filtering is done on the vlan interfaces. Normally we have around 10-20 vlans on each machine, and we have a LOT of rules in pf.conf. To make configuration a little easier I'm beginning to look at how to separate the vlans into multiple configs, one for each vlan, and then include them all from pf.conf. I would want to have all macros, options and rules for each vlan in a separate file, but also i would like to use macros from one config in rules in another file. To clarify what I'm getting at, here's an example: ## /etc/vlan500.conf: DB="192.168.0.10/32" block log on vlan500 pass in quick on vlan500 from $Webserver to $DB port 3306 pass out on vlan500 ## /etc/vlan1000.conf: Webserver="192.168.1.20/32" block log on vlan1000 pass in quick on vlan1000 from any to $Webserver port 80 pass out on vlan1000 ## /etc/pf.conf include "/etc/vlan500.conf" include "/etc/vlan1000.conf" ## The above example would not work, as pfctl will look at the rules in vlan500.conf before looking at the macros in vlan1000.conf and it will throw an error that the $Webserver macro is not defined. If I change the order of the includes in pf.conf, it will work, but of course of I try to use macros from vlan1000.conf for rules in vlan500.conf, the problem will arise again. One way to solve it would be to put all the macros in, say, /etc/vlan500-macros.conf and /etc/vlan1000-macros.conf and make sure they are included before the rules in pf.conf, but that seems inconvenient to me. What is the common practice for using includes? Is there a way to get pfctl to read ALL macros from ALL files before looking at the rules? I would be happy to hear some suggestions. Thanks, Peter
Re: pf and includes
On 2011-11-30 16:14, Guido Tschakert wrote: > > How about a definition.conf with all your (Name,IP-Adress)-Pairs which > is included first in your pf.conf, so your vlan.confs only include > the rules but no definitions. > > guido > Thanks, this is probably the way to do it. Sometimes we move vlans between firewalls and then it can be good to remove the rules, but still keep some macros. I'm also planning to have the same set of variables on all 10 firewalls so that the only difference between them will be the rules files. //Peter
Re: pf and includes
On 2011-11-30 20:20, Adriaan wrote: > > You could use a Makefile to concatenate a pf.conf from separate files. > This can give more flexibility than provided by "include" : Thank you very much for your elaborate solution. To keep things a little less complex, I will probably go with includes and keep all the macros and tables in one big file shared on all firewalls. //Peter
"block return" on bridge(4)
Hello, >From man pf.conf: "Options returning ICMP packets currently have no effect if pf(4) operates on a bridge(4), as the code to support this feature has not yet been implemented." Just wondering, will this be implemented? If I understand correctly, if "block return" is set on a bridging firewall TCP RST will be sent out when TCP is blocked, but nothing is sent out when UDP or any other protocol is blocked. Right? Thanks, Peter Hallin, Lund University
Re: Support for Intel X520-T2 10GbaseT cards
On 2011-06-16 14:43, Peter Hallin wrote: > On 2011-06-15 22:40, Jonathan Gray wrote: > > > > Yes this seems low indeed. You could try use rdomains and route exec > > with a cable between the two ports to make packets go over the interfaces > > if you don't have another host with 10G interfaces. > > I didn't get that far. When connecting the two ports to each other (tried > with a couple of TP cables) I only get 1000baseT full-duplex on ix0 and > ix1. > > > > > If you've had ppb problems with em I'd imagine they would affect ix as well > > if it is sitting behind the same ppb. > > I disabled the ppb hotplug interrupt in ppb.c as Claudio mentioned in > "Performance degradation after upgrade" and ran the same tests again. > > This time it got up to 932 Mbit/s (close to theoretical maximum), but > the CPU usage was 100% intrs. However, the interfaces only generated > around 2000 intrs/s each (compared to 13000 in my last test). > > I will not have more time to do additional testing in a while as i'm off > for vacation, but I hope my tests so far can give you some insight. > > Thanks, Peter > Hello, I finally got some time to test this 10G stuff again and now I'm using two machines with the same cards running 5.0 RELEASE (amd64/sp). Problem is as before, when connected and in UP state, the media goes down to 1000BaseT. If I take them both down, it goes back up to 10GbaseT. Weird. Tested with different (brand new cables) both cat 5E and cat 6. Good news is that the interrupt rate seems to be better, only about 15% CPU when running at 1Gbit/s and around 1500 intrs/s. These are two identical machines with the same hardware. //Peter dmesg: OpenBSD 5.0 (GENERIC) #53: Wed Aug 17 10:07:52 MDT 2011 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC RTC BIOS diagnostic error 80 real mem = 3208368128 (3059MB) avail mem = 3108941824 (2964MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xe4380 (23 entries) bios0: vendor Intel Corp. version "BPX5810J.86B.0034.2010.0414.1511" date 04/14/2010 bios0: Intel Corporation WX58BP acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S3 S4 S5 acpi0: tables DSDT FACP APIC WDDT MCFG ASF! HPET ASPT WDTx SSDT DMAR acpi0: wakeup devices SLPB(S4) P32_(S4) ECIR(S4) UAR1(S4) ILAN(S4) PEG1(S4) PEG3(S4) PEG7(S4) PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) PEX4(S5) PEX5(S5) UHC1(S3) UHC2(S3) UHC3(S3) UHC4(S3) EHCI(S3) EHC2(S3) UH42(S3) UHC5(S3) UHC6(S3) AZAL(S3) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU W3690 @ 3.47GHz, 3467.31 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,PCLMUL,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG cpu0: 256KB 64b/line 8-way L2 cache cpu0: apic clock running at 133MHz ioapic0 at mainbus0: apid 8 pa 0xfec0, version 20, 24 pins acpimadt0: unknown apic structure type 7f acpimcfg0 at acpi0 addr 0xf800, bus 0-63 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 10 (P32_) acpiprt2 at acpi0: bus 1 (PEG1) acpiprt3 at acpi0: bus 5 (PEG3) acpiprt4 at acpi0: bus 6 (PEG7) acpiprt5 at acpi0: bus 7 (PEX0) acpiprt6 at acpi0: bus 8 (PEX1) acpiprt7 at acpi0: bus -1 (PEX3) acpiprt8 at acpi0: bus 9 (PEX4) acpicpu0 at acpi0: C1, PSS acpibtn0 at acpi0: SLPB cpu0: Enhanced SpeedStep 3467 MHz: speeds: 3459, 3458, 3325, 3192, 3059, 2926, 2793, 2660, 2527, 2394, 2261, 2128, 1995, 1862, 1729, 1596 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel X58 Host" rev 0x13 ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13: msi pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0c pci2 at ppb1 bus 2 ppb2 at pci2 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0c pci3 at ppb2 bus 3 em0 at pci3 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: msi, address 00:1b:21:c8:8f:ac em1 at pci3 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: msi, address 00:1b:21:c8:8f:ad ppb3 at pci2 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0c pci4 at ppb3 bus 4 em2 at pci4 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: msi, address 00:1b:21:c8:8f:ae em3 at pci4 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: msi, address 00:1b:21:c8:8f:af ppb4 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13: msi pci5 at ppb4 bus 5 vga1 at pci5 dev 0 function 0 "NVIDIA GeForce 210" rev 0xa2 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) azalia0 at pci5 dev 0 function 1 vendor "NVIDIA", unknown product 0x0be3 rev 0xa1:
Support for Intel X520-T2 10GbaseT cards
Hi all, After reading the manpages for ix(4) "Intel 82598/82599 PCI Express 10Gb Ethernet device", I drew the conclusion that the X520-T2 (http://www.intel.com/Products/Server/Adapters/esa-x520-t2/ethernet-esa-x520- t2-overview.htm) would also be supported by the driver, so we took a shot and bought a pair for our firewalls. It is based on the 82599 chipset, so why wouldn't it? My mistake, is seems.. I get an "unknown product 0x151c" error in dmesg (see below). So now we have a pair of these cards and the obvious question is of course, will they be supported? I realize that I should have checked this before I bought it, but as Intel claims in the specs (http://www.intel.com/Assets/PDF/prodbrief/318349-004.pdf) it is supported by FreeBSD and I believe that's from where the driver was ported. I'm very grateful for any input. Peter Hallin Lund University dmesg: OpenBSD 4.8 (GENERIC.MP) #335: Mon Aug 16 09:09:20 MDT 2010 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP RTC BIOS diagnostic error 2 real mem = 2139598848 (2040MB) avail mem = 2068819968 (1972MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7fa32000 (60 entries) bios0: vendor Intel Corporation version "S5000.86B.11.00.0096.011320091422" date 01/13/2009 bios0: Intel S5000VSA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET MCFG SSDT SSDT SSDT HEST BERT ERST EINJ acpi0: wakeup devices SLPB(S5) P32_(S5) PS2M(S1) PS2K(S1) UAR1(S5) UAR2(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S5) PCIO(S5) PCIP(S5) PCIQ(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.08 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu0: 6MB 64b/line 16-way L2 cache cpu0: apic clock running at 332MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu1: 6MB 64b/line 16-way L2 cache cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu2: 6MB 64b/line 16-way L2 cache cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3 ,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu3: 6MB 64b/line 16-way L2 cache ioapic0 at mainbus0: apid 8 pa 0xfec0, version 20, 24 pins ioapic1 at mainbus0: apid 9 pa 0xfec8, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 9 (P32_) acpiprt2 at acpi0: bus 8 (PEX0) acpiprt3 at acpi0: bus -1 (PEX1) acpiprt4 at acpi0: bus -1 (PEX2) acpiprt5 at acpi0: bus -1 (PEX3) acpiprt6 at acpi0: bus 1 (PCIE) acpiprt7 at acpi0: bus 6 (PCIE) acpiprt8 at acpi0: bus 2 (PCIW) acpiprt9 at acpi0: bus 3 (PCIO) acpiprt10 at acpi0: bus 4 (PCIP) acpiprt11 at acpi0: bus 5 (PCIQ) acpicpu0 at acpi0: C2, C1, PSS acpicpu1 at acpi0: C2, C1, PSS acpicpu2 at acpi0: C2, C1, PSS acpicpu3 at acpi0: C2, C1, PSS acpibtn0 at acpi0: SLPB ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2493 MHz: speeds: 2497, 1998 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 5000V Host" rev 0xb1 ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0xb1 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci2 at ppb1 bus 2 ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 16 (irq 10) pci3 at ppb2 bus 3 vendor "Intel", unknown product 0x151c (class network subclass ethernet, rev 0x01) at pci3 dev 0 function 0 not configured vendor "Intel", unknown product 0x151c (class network subclass ethernet, rev 0x01) at pci3 dev 0 function 1 not configured ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 17 (irq 11) pci4 at ppb3 bus 4 vendor "Intel", unknown product 0x151c (class network subclass ethernet, rev 0x01) at pci4 dev 0 function 0 not configured vendor "Intel", unknown product 0x151c (class network subclass ethernet, rev 0x01) at pci4 dev 0 function 1 not configured ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE&q
Re: Support for Intel X520-T2 10GbaseT cards
tion 0 "Intel 5000 PCIE x8" rev 0xb1 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci2 at ppb1 bus 2 ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 16 (irq 10) pci3 at ppb2 bus 3 ix0 at pci3 dev 0 function 0 "Intel 82599 T3" rev 0x01: apic 8 int 17 (irq 11), address 00:1b:21:93:a6:3c ix1 at pci3 dev 0 function 1 "Intel 82599 T3" rev 0x01: apic 8 int 16 (irq 10), address 00:1b:21:93:a6:3d ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 17 (irq 11) pci4 at ppb3 bus 4 ix2 at pci4 dev 0 function 0 "Intel 82599 T3" rev 0x01: apic 8 int 18 (irq 5), address 00:1b:21:93:a6:40 ix3 at pci4 dev 0 function 1 "Intel 82599 T3" rev 0x01: apic 8 int 17 (irq 11), address 00:1b:21:93:a6:41 ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01 pci5 at ppb4 bus 5 em0 at pci5 dev 0 function 0 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 18 (irq 5), address 00:15:17:89:eb:fc em1 at pci5 dev 0 function 1 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 19 (irq 11), address 00:15:17:89:eb:fd ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01 pci6 at ppb5 bus 6 ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1 pci7 at ppb6 bus 7 "Intel I/OAT" rev 0xb1 at pci0 dev 8 function 0 not configured pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0xb1 pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0xb1 pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0xb1 pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0xb1 pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0xb1 pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0xb1 pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0xb1 ppb7 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09: apic 8 int 16 (irq 10) pci8 at ppb7 bus 8 uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 22 (irq 5) uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09: apic 8 int 22 (irq 5) ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23 (irq 11) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb8 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9 pci9 at ppb8 bus 9 vga1 at pci9 dev 12 function 0 "ATI ES1000" rev 0x02 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) radeondrm0 at vga1: apic 8 int 17 (irq 11) drm0 at radeondrm0 pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09 pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 pciide0: channel 1 disabled (no drives) pciide1 at pci0 dev 31 function 2 "Intel 6321ESB SATA" rev 0x09: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide1: using apic 8 int 20 (irq 10) for native-PCI interrupt atapiscsi0 at pciide1 channel 0 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: ATAPI 5/cdrom removable cd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5 ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int 20 (irq 10) iic0 at ichiic0 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb2 at uhci1: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci2: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci3: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 mtrr: Pentium Pro MTRR support softraid0 at root root on wd0a swap on wd0b dump on wd0b On Tue, Mar 15, 2011 at 05:02:38PM +0100, Peter Hallin wrote: > Hi all, > > After reading the manpages for ix(4) "Intel 82598/82599 PCI Express 10Gb > Ethernet device", > I drew the conclusion that the X520-T2 > (http://www.intel.com/Products/Server/A
Re: Support for Intel X520-T2 10GbaseT cards
Thanks, I tried that and got the routing table, however still no packets coming through. //Peter > sounds like rDNS delay. > > retry with arp -an and netstat -rn > > /Pete
Performance degradation after upgrade
Hello all, Last saturday during a service window, we installed a new firewall as a replacement to one of our oldest firewalls that had been running at a very high load for a long time. The old fw ran 3.9 (i386, SP) on a Pentium4 3.2Ghz and was able to forward data at about 300 Mbit/s at 100% CPu usage (interrupts). The NICs in this machine was Intel 1000/Pro cards on the PCI bus. We were hoping that the new hardware would be able to achieve a higher throughput, perhaps near 1 Gbit/s. The result, however was even worse than the old fw. We can only get about 70-80 Mbit/s at 100% CPU... The new system has a Xeon 4-core CPU at 2,5 Ghz and Intel 1000/Pro cards on the PCI-E bus. It's running 4.8 GENERIC.MP amd64. What we noticed when checking systat was a high number of livelocks on the fw interfaces, em0 and em2: IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM System25650 458 2k302142 lo0 em0 3133902k 84 256 8 em1 em2 3790512k11 4 25611 em3 em449622k 4 4 256 4 This seems extremely high compared to some of our other fws where the number of livelocks are a couple of hundreds after months of uptime. What could be the reason for this huge drop in performance? I know that the kernel only uses on CPU core and the clock frequency is a little lower, but it seems the difference is too high. Could it be the PCI-E NICs? The bus speed is much faster than the old PCI cards, so we hoped they would perform better. Could it be the OS? 64 bits instead of 32 and MP kernel instead of SP? What we do know is that it's not PF, as we disabled it for a couple of minutes and still had 100% interrupts. Do any of you have the same issues with high interrupts and low throughput? We really don't know where to start... :( I'm very grateful for any kind of input regarding this matter. Brgds, Peter Hallin, Lund University Sweden dmesg: OpenBSD 4.8 (GENERIC.MP) #335: Mon Aug 16 09:09:20 MDT 2010 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP RTC BIOS diagnostic error 2 real mem = 2139598848 (2040MB) avail mem = 2068819968 (1972MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7fa32000 (60 entries) bios0: vendor Intel Corporation version "S5000.86B.11.00.0096.011320091422" date 01/13/2009 bios0: Intel S5000VSA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET MCFG SSDT SSDT HEST BERT ERST EINJ acpi0: wakeup devices SLPB(S5) P32_(S5) PS2M(S1) PS2K(S1) UAR1(S5) UAR2(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S5) PCIO(S5) PCIP(S5) PCIQ(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.09 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu0: 6MB 64b/line 16-way L2 cache cpu0: apic clock running at 332MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu1: 6MB 64b/line 16-way L2 cache cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu2: 6MB 64b/line 16-way L2 cache cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu3: 6MB 64b/line 16-way L2 cache ioapic0 at mainbus0: apid 8 pa 0xfec0, version 20, 24 pins ioapic1 at mainbus0: apid 9 pa 0xfec8, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz
Re: Performance degradation after upgrade
This is the output when the machine is running at 80 Mbit/s and CPU usage is almost 100% interrupts: Please note that this is after we rebooted with the SP kernel, which didn't make any differences. systat ifs: IFACE STATE DESC IPKTS IBYTESIERRSOPKTS OBYTESOERRSCOLLS em0up:U 6772 82850950 5043 125182100 em1dn:D 0000 000 em2up:U 5050 12685740 6762 828026200 em3dn:D 0000 000 em4up:U 19 71360 774 3616390 vmstat -iz: interrupt total rate irq0/clock 73974 100 irq144/acpi000 irq112/ppb2 00 irq112/em0 767598 1038 irq113/em1 00 irq113/ppb3 00 irq113/em21007863 1363 irq114/em3 00 irq114/em4 460696 623 irq115/em5 00 irq112/ppb7 00 irq96/uhci0 00 irq97/uhci1 00 irq96/uhci2 00 irq97/uhci3 00 irq96/ehci0 00 irq98/pciide020832 irq99/pciide1 00 irq99/ichiic0 00 irq145/com0 00 irq146/com1 00 irq147/pckbc0 3040 irq148/pckbc0 00 Total 2312518 3129 The next step for us will be to swap the PCIe NICs to regular PCI to see if that does anything. On 2011-03-28 13:55, Jonathan Gray wrote: > It sounds like you have an interrupt storm what does > vmstat -iz show?
Re: Performance degradation after upgrade
I realized now that this measurement is wrong. "vmstat -iz" seems to calculate the interrupt rate based a longer period, and this measurement was taken just after we started to push traffic through the machine again. The amount of interrupts per second when checking with systat (which seems to have a shorter measurement period) at the same time was way higher, about 5000 intr/s on em0 and em2. Sorry for the wrong data On 2011-03-28 17:46, Peter Hallin wrote: > This is the output when the machine is running at 80 Mbit/s > and CPU usage is almost 100% interrupts: > > Please note that this is after we rebooted with the SP kernel, > which didn't make any differences. > > systat ifs: > > IFACE STATE DESC IPKTS IBYTESIERRSOPKTS > OBYTESOERRSCOLLS > em0up:U 6772 82850950 5043 > 125182100 > em1dn:D 0000 >000 > em2up:U 5050 12685740 6762 > 828026200 > em3dn:D 0000 >000 > em4up:U 19 71360 774 > 3616390 > > vmstat -iz: > > interrupt total rate > irq0/clock 73974 100 > irq144/acpi000 > irq112/ppb2 00 > irq112/em0 767598 1038 > irq113/em1 00 > irq113/ppb3 00 > irq113/em21007863 1363 > irq114/em3 00 > irq114/em4 460696 623 > irq115/em5 00 > irq112/ppb7 00 > irq96/uhci0 00 > irq97/uhci1 00 > irq96/uhci2 00 > irq97/uhci3 00 > irq96/ehci0 00 > irq98/pciide020832 > irq99/pciide1 00 > irq99/ichiic0 00 > irq145/com0 00 > irq146/com1 00 > irq147/pckbc0 3040 > irq148/pckbc0 00 > Total 2312518 3129 > > > The next step for us will be to swap the PCIe NICs to > regular PCI to see if that does anything. > > > On 2011-03-28 13:55, Jonathan Gray wrote: > > It sounds like you have an interrupt storm what does > > vmstat -iz show? > -- -- Peter Hallin | Tel: +46 46 2221341 IT-Sdkerhet och brandvdggar| Fax: +46 46 2221399 LDC, Lunds Universitet | E-post: peter.hal...@ldc.lu.se Margaretav. 1A, 222 40, LUND | http://www.ldc.lu.se/security/
Re: Performance degradation after upgrade
Ok, now we have been doing some testing and probably found the problem. All tests were done on the same machine with an Intel S5000VSA MB and a Xeon E5420 2,5 Ghz processor, running OpenBSD 4.8 amd64 GENERIC (SP kernel). We tested the performance with iperf, running two clients connected through a bridge. With the Intel Pro/1000 PCIe (82576) dual port cards with the bridge between two cards (in this case em0 to em2) we got the worst performance, 150 Mbit/s. While testing this and watching with 'systat vmstat', the CPU was 99% busy handling interrupts. The amount of interrupts were about 3000/s on em0 (where the iperf client was connected) and 1500/s on em2 (iperf server). At the same time 'systat ifs' showed about 10 new livelocks per second. Next we tested regular PCI Intel Pro/1000MT (82545GM) cards and now we got the performance we had hoped for in the first place. 910 Mbit/s with 8000 intr/s on both cards at 50% CPU (intr). No livelocks. We thought perhaps the issue was related to the PCIe bus, so we did one final test, this time with quad port Intel Pro/1000 QP (82576) PCIe cards. These performed excellent, with 940 Mbit/s, 8200 intr/s per card and 60% CPU (intr). So, it seems the dual port PCIe cards suck and we have to replace them. //Peter On 2011-03-29 07:40, Peter Hallin wrote: > I realized now that this measurement is wrong. > > "vmstat -iz" seems to calculate the interrupt rate based a longer > period, and this measurement was taken just after we started to push > traffic through the machine again. > > The amount of interrupts per second when checking with systat (which > seems to have a shorter measurement period) at the same time was way > higher, about 5000 intr/s on em0 and em2. > > Sorry for the wrong data
Re: Performance degradation after upgrade
On 2011-03-30 21:18, Rodrigo Mosconi wrote: > > > > Just as curiosity: > > Did you used both ports from the Intel Pro/1000 PCIe (82576)? > > And if is used a single port PCI-Ex Intel Card? > This is what we have tested today: 1. One dual port PCIe, with port 1 (em0) bridged with port 2 (em1), with bad results. 2. Two dual port PCIe, with port 1 on the first card (em0) bridged with port 1 (em2) on the second card. Same results. 3. One dual port PCIe, with port 1 (em0) bridged with the motherboard's built-in card (em2). Better results, but way too bad... ;) We don't have any single port PCIe cards, so that we haven't tested.
Re: Performance degradation after upgrade
On 2011-03-30 14:27, Claudio Jeker wrote: > Could you donate a dual port card to the project if you replace them? > I would like to figure out why some em(4) perform badly while the same > chip on a different card seems to perform as expected. > > Can you provide the vmstat -zi output of the 4 port card? I wonder how > the interrupts are shared on the 2 port and the 4 port card. IIRC on the > original vmstat -zi output em0 shared the interrupt with the pci bridge. > > -- > :wq Claudio I have emailed you about the possibilities to donate a card. We did some more tests today, using only ONE dual port card in the machine, but the results were the same. Below is dmesg and vmstat -iz for both the 4-port and the 2-port cards. 4-port card, one card installed in machine: dmesg: OpenBSD 4.8 (GENERIC) #182: Mon Aug 16 09:02:40 MDT 2010 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC RTC BIOS diagnostic error 3 real mem = 1065857024 (1016MB) avail mem = 1023684608 (976MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x3fa32000 (58 entries) bios0: vendor Intel Corporation version "S5000.86B.11.00.0096.011320091422" date 01/13/2009 bios0: Intel S5000VSA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET MCFG SSDT SSDT SSDT HEST BERT ERST EINJ acpi0: wakeup devices SLPB(S5) P32_(S5) PS2M(S1) PS2K(S1) UAR1(S5) UAR2(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S5) PCIO(S5) PCIP(S5) PCIQ(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.04 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu0: 6MB 64b/line 16-way L2 cache cpu0: apic clock running at 332MHz cpu at mainbus0: not configured cpu at mainbus0: not configured cpu at mainbus0: not configured ioapic0 at mainbus0: apid 8 pa 0xfec0, version 20, 24 pins ioapic1 at mainbus0: apid 9 pa 0xfec8, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 12 (P32_) acpiprt2 at acpi0: bus 11 (PEX0) acpiprt3 at acpi0: bus -1 (PEX1) acpiprt4 at acpi0: bus -1 (PEX2) acpiprt5 at acpi0: bus -1 (PEX3) acpiprt6 at acpi0: bus 1 (PCIE) acpiprt7 at acpi0: bus 9 (PCIE) acpiprt8 at acpi0: bus 2 (PCIW) acpiprt9 at acpi0: bus 3 (PCIO) acpiprt10 at acpi0: bus 4 (PCIP) acpiprt11 at acpi0: bus 8 (PCIQ) acpicpu0 at acpi0: C2, C1, PSS acpibtn0 at acpi0: SLPB ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2494 MHz: speeds: 2497, 1998 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 5000V Host" rev 0xb1 ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0xb1 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci2 at ppb1 bus 2 ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 16 (irq 10) pci3 at ppb2 bus 3 ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 17 (irq 11) pci4 at ppb3 bus 4 ppb4 at pci4 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e pci5 at ppb4 bus 5 ppb5 at pci5 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e pci6 at ppb5 bus 6 em0 at pci6 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 8 int 19 (irq 5), address 00:1b:21:81:57:68 em1 at pci6 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 8 int 16 (irq 10), address 00:1b:21:81:57:69 ppb6 at pci5 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e pci7 at ppb6 bus 7 em2 at pci7 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 8 int 17 (irq 11), address 00:1b:21:81:57:6c em3 at pci7 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 8 int 18 (irq 11), address 00:1b:21:81:57:6d ppb7 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01 pci8 at ppb7 bus 8 em4 at pci8 dev 0 f
Re: Performance degradation after upgrade
OK, here's a little update on this problem. As I told you earlier in the thread, we did some successful tests with the 4-port Intel 82576 card, HOWEVER we only tested two ports, em0 och em1. When the card later was put into the production machine we chose to use em0 as the unprocteded if and em2 as the protected. When we connected the machine to the router, we only got a throughput at about 100 Mbit/s 100% CPU, busy handling interrupts, while the cards only generate about 3000 intr/s each. As it turns out, the problem is the same as with the dual port cards, but now it only applies to em2 and em3. If we any of those ports, the machine get bogged down completely and the livelock rate is about 10 per second. If we only use em0 and em1, it performs really good. With real traffic, we've had some peaks at about 750 Mbit/s at 80% CPU and when testing with iperf we get 936 Mbit/s at 90%i CPU (8200 intr/s). This leads me to believe that there are some IRQ conflicts that may or may not be caused by the driver. This is the vmstat -iz output of the production machine: interrupt total rate irq0/clock 743258 99 irq144/acpi000 irq114/ppb2 00 irq112/em0 52133849 7004 irq113/em1 52036815 6991 irq114/em2 00 irq115/em3 00 irq115/ppb6 00 irq112/em4 413195 irq113/em5 00 irq114/ppb1000 irq96/uhci0 00 irq97/uhci1 00 irq96/uhci2 00 irq97/uhci3 00 irq96/ehci0 00 irq98/pciide081621 irq99/pciide1 00 irq99/ichiic0 00 irq145/com0 00 irq146/com1 00 irq147/pckbc0 6450 irq148/pckbc0 00 Total 10496404814102 Brgds, Peter
Re: Performance degradation after upgrade
On 2011-04-05 14:35, Claudio Jeker wrote: > Can you give the following diff a spin and see if that makes the card act > faster. This disables the ppb hotplug interrupt which is shared with the > em2 and em3 interrupts. > > -- > :wq Claudio Ok, that did the trick. I made the changes to the 4.8 source and ppb hotplug was disabled. I then tested the dual port cards and got close to 1 Gbit/s but without the high CPU usage (only about 30% intr). So my question now is: Do we need the ppb hotplug? What is it good for? //Peter
Re: Performance degradation after upgrade
On 2011-04-06 16:43, Claudio Jeker wrote: > > Wait. It seems more is needed. Will come back when we have a better > solution. > Alright. Your first quick fix is good enough for us, we don't use expresscards in our firewalls.. ;) I actually tested it on an older 4.4 fw that has been under heavy load despite low traffic, and it was a drastic improvement. When you are ready with a new diff against -current, I'll be happy to test it with our stuff. Until then, we are satisfied with disabling ppb hotplug completely. //Peter
Re: Support for Intel X520-T2 10GbaseT cards
On 2011-06-15 04:26, Jonathan Gray wrote: > Claudio has recently synced ix to a newer version of the Intel code, > if you can try again with -current this should work. Great, thanks. I tried it out today, but the performance wasn't exactly what I had expected. This is a dual port card, so I bridged ix0 with ix1 and ran iperf between two hosts through the card. The hosts were connected with 1Gbase as I have no other 10Gbase cards. It only got up to about 450 Mbit/s (standard TCP iperf test) and there was an unbalance between ix0 and ix1 when it came to interrupts. ix1 only got up to 4000 intr/s whereas ix1 got up to 13000 intr/s (when monitoring live with systat vmstat). The CPU intr usage was around 50-60%. Could this possibly be related to the ppb problems we've seen with em(4) cards? //Peter vmstat -iz: interrupt total rate irq0/clock 345997 100 irq144/acpi000 irq96/ppb2 00 irq97/ix0 1046011 302 irq96/ix1 327045 94 irq97/ppb3 00 irq112/em0 52826 15 irq113/em1 00 irq96/ppb7 00 irq98/uhci0 4440 irq99/uhci1 00 irq98/uhci2 00 irq99/uhci3 00 irq98/ehci0 20 irq100/pciide0 14620 irq101/ahci010 irq101/ichiic0 00 irq145/com0 00 irq146/com1 00 irq147/pckbc0 00 Total 1773788 513 dmesg: OpenBSD 4.9-current (GENERIC) #0: Wed Jun 15 10:51:41 CEST 2011 root@lab:/usr/src/sys/arch/amd64/compile/GENERIC RTC BIOS diagnostic error 3 real mem = 1065857024 (1016MB) avail mem = 1023438848 (976MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x3fa32000 (58 entries) bios0: vendor Intel Corporation version "S5000.86B.11.00.0096.011320091422" date 01/13/2009 bios0: Intel S5000VSA acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET MCFG SSDT SSDT SSDT HEST BERT ERST EINJ acpi0: wakeup devices SLPB(S5) P32_(S5) PS2M(S1) PS2K(S1) UAR1(S5) UAR2(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S5) PCIO(S5) PCIP(S5) PCIQ(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.10 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG cpu0: 6MB 64b/line 16-way L2 cache cpu0: apic clock running at 332MHz cpu at mainbus0: not configured cpu at mainbus0: not configured cpu at mainbus0: not configured ioapic0 at mainbus0: apid 8 pa 0xfec0, version 20, 24 pins ioapic1 at mainbus0: apid 9 pa 0xfec8, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpimcfg0 at acpi0 addr 0xa000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 9 (P32_) acpiprt2 at acpi0: bus 8 (PEX0) acpiprt3 at acpi0: bus -1 (PEX1) acpiprt4 at acpi0: bus -1 (PEX2) acpiprt5 at acpi0: bus -1 (PEX3) acpiprt6 at acpi0: bus 1 (PCIE) acpiprt7 at acpi0: bus 6 (PCIE) acpiprt8 at acpi0: bus 2 (PCIW) acpiprt9 at acpi0: bus 3 (PCIO) acpiprt10 at acpi0: bus 4 (PCIP) acpiprt11 at acpi0: bus 5 (PCIQ) acpicpu0 at acpi0: C2, C1, PSS acpibtn0 at acpi0: SLPB ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2494 MHz: speeds: 2497, 1998 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 5000V Host" rev 0xb1 ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0xb1 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci2 at ppb1 bus 2 ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 16 pci3 at ppb2 bus 3 ix0 at pci3 dev 0 function 0 "Intel 82599 T3" rev 0x01: apic 8 int 17, address 00:1b:21:93:a6:3c ix1 at pci3 dev 0 function 1 "Intel 82599 T3" rev 0x01: apic 8 int 16, address 00:1b:21:93:a6:3d ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01: apic 8 int 17 pci4 at ppb3 bus 4 ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01 pci5 at ppb4 bus 5 em0 at pci5 dev 0 function 0 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 18, address 00:15:17:60:97:fc em1 at pci5 dev 0 function 1 "Intel PRO/1000 PT (80003ES2)" rev 0x01: apic 8 int 19, address 00:15:17:60:97:fd ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01 pci6 at ppb5 bus 6 ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1 pci7 at ppb6 bus 7 pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting
Re: Support for Intel X520-T2 10GbaseT cards
On 2011-06-15 13:58, FRLinux wrote: > Just out of curiosity, which iperf settings did you use? Just standard TCP, iperf -c hostname > > I know this is on a Linux box, but just out of interest, I get full > speed there: Yeah, sure it works on Linux, but that's not what I'm running. Otherwise it's almost the same card as yours, but with TP connectors. //peter > > iperf -c xxx.xxx.xxx.xxx -i 5 > > Client connecting to xxx.xxx.xxx.xxx, TCP port 5001 > TCP window size: 16.0 KByte (default) > > [ 3] local xxx.xxx.xxx.xxx port 47035 connected with xxx.xxx.xxx.xxx port > 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0- 5.0 sec 5.47 GBytes 9.39 Gbits/sec > [ 3] 0.0-10.0 sec 10.9 GBytes 9.40 Gbits/sec > > Cards are: > Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) > driver: ixgbe > version: 2.0.44-k2 > firmware-version: 0.9-3 > bus-info: :05:00.0 > > Cheers, > Steph
Re: Support for Intel X520-T2 10GbaseT cards
On 2011-06-15 22:40, Jonathan Gray wrote: > > Yes this seems low indeed. You could try use rdomains and route exec > with a cable between the two ports to make packets go over the interfaces > if you don't have another host with 10G interfaces. I didn't get that far. When connecting the two ports to each other (tried with a couple of TP cables) I only get 1000baseT full-duplex on ix0 and ix1. > > If you've had ppb problems with em I'd imagine they would affect ix as well > if it is sitting behind the same ppb. I disabled the ppb hotplug interrupt in ppb.c as Claudio mentioned in "Performance degradation after upgrade" and ran the same tests again. This time it got up to 932 Mbit/s (close to theoretical maximum), but the CPU usage was 100% intrs. However, the interfaces only generated around 2000 intrs/s each (compared to 13000 in my last test). I will not have more time to do additional testing in a while as i'm off for vacation, but I hope my tests so far can give you some insight. Thanks, Peter
Re: Support for Intel X520-T2 10GbaseT cards
On 2011-06-16 14:43, Peter Hallin wrote: > > I didn't get that far. When connecting the two ports to each other (tried > with a couple of TP cables) I only get 1000baseT full-duplex on ix0 and > ix1. Nevermind that. After a reboot with the cables connected the interfaces show up as 10GbaseT full-duplex. Will see if I have the time to do some tests. //Peter
