On Wed, Aug 07, 2013 at 10:26:22AM -0400, Maxim Khitrov wrote:
> Hi all,
>
> I'm looking for performance measuring and tuning advice for 10 gigabit
> Ethernet. I have a pair of Lanner FW-8865 systems that will be used as
> firewalls for the local network.
[...]
> The initial iperf runs couldn't go beyond ~3.2 Gbps:
you expect a lot of localy generated traffic on your firewall?
(if the answer is no, why are you testing that?)
[...]
> Increasing the MTU on both ix0 interfaces to 9000 gives me ~7.2 Gbps:
you expect a lot of jumbo frames in front of / behind your firewall?
(if the answer is no, why are you testing that?)
anyway, I was testing an Intel 82599 system in July which will become
a border router. All of this is forwarding rate; it took me 2 days to
beg, borrow and steal enough hw to actually generate the traffic. (I
had 4 systems in front of and 4 systems behind the router, all doing
1Gb/s)
kern.pool_debug=1, no pf
6.62 Gb/s, 636946 P/s, 21221 intr/s
kern.pool_debug=0, no pf
7.39 Gb/s, 706994 P/s, 16535 intr/s
kern.pool_debug=0, pf enabled, all relevant rules have "no state"
1.32 Gb/s, 139587 P/s, 15357 intr/s
kern.pool_debug=0, pf enabled, all relevant rules have "keep state (sloppy)"
4.82 Gb/s, 470514.87 P/s, 31318 intr/s
regarding the no state and keep state (sloppy) tests: as said, this
will be a border router and _will_ see asymetric routing. I expect a
"normal" pf.conf i.e. with keep state to be on the order of keep state
(sloppy); I did not test that though.
also keep in mind that this is a very very rough benchmark, essentialy
it estimates an upper limit. I don't expect that in production there
will be 8 flows with maximum speed and tens of states in pf but rather
thousands of flows and hunderedthousands of states.
OpenBSD 5.3-current (GENERIC.MP) #18: Sat Jul 6 16:54:29 MDT 2013
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 38629015552 (36839MB)
avail mem = 37592993792 (35851MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x99c00 (100 entries)
bios0: vendor American Megatrends Inc. version "2.1b" date 11/30/2012
bios0: Supermicro X8DTU-LN4+
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG SLIT SPMI OEMB HPET SSDT EINJ BERT ERST HEST
acpi0: wakeup devices NPE1(S4) NPE2(S4) NPE3(S4) NPE4(S4) NPE5(S4) NPE6(S4)
NPE7(S4) NPE8(S4) NPE9(S4) NPEA(S4) P0P1(S4) USB0(S4) USB1(S4) USB2(S4)
USB5(S4) EUSB(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, 3200.46 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: apic clock running at 133MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, 2666.76 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 18 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, 2666.76 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 9, package 0
cpu3 at mainbus0: apid 20 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5640 @ 2.67GHz, 2666.76 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AES,NXE,LONG,LAHF,PERF,ITSC
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 10, package 0
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 4, remapped to apid 1
ioapic1 at mainbus0: apid 3 pa 0xfec8a000, version 20, 24 pins
ioapic1: misconfigured as apic 6, remapped to apid 3
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (NPE1)
acpiprt2 at acpi0: bus -1 (NPE2)
acpiprt3 at acpi0: bus 2 (NPE3)
acpiprt4 at acpi0: bus -1 (NPE4)
acpiprt5 at acpi0: bus 3 (NPE5)
acpiprt6 at acpi0: bus -1 (NPE6)
acpiprt7 at acpi0: bus 4 (NPE7)
acpiprt8 at acpi0: bus -1 (NPE8)
acpiprt9 at acpi0: bus -1 (NPE9)
acpiprt10 at acpi0: bus -1 (NPEA)
acpiprt11 at acpi0: bus 7 (P0P1)
acpiprt12 at acpi0: bus -1 (P0P5)
acpiprt13 at acpi0: bus -1 (P0P6)
acpiprt14 at acpi0: bus -1 (P0P7)
acpiprt15 at acpi0: bus -1 (P0P9)
acpiprt16 at acpi0: bus 5 (P0P4)
acpiprt17 at acpi0: bus 6 (P0P8)
acpicpu0 at acpi0: C3, C2, C1, PSS
acpicpu1 at acpi0: C3, C2, C1, PSS
acpicpu2 at acpi0: C3, C2, C1, PSS
acpicpu3 at acpi0: C3, C2, C1, PSS
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 3200 MHz: speeds: 2668, 2667, 2533, 2400, 2267, 2133,
2000, 1867, 1733, 1600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5520 Host" rev 0x22
ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x22: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x22: msi
pci2 at ppb1 bus 2
ppb2 at pci0 dev 5 function 0 "Intel X58 PCIE" rev 0x22: msi
pci3 at ppb2 bus 3
ppb3 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x22: msi
pci4 at ppb3 bus 4
ix0 at pci4 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
90:e2:ba:12:84:68
ix1 at pci4 dev 0 function 1 "Intel 82599" rev 0x01: msi, address
90:e2:ba:12:84:69
"Intel X58 IOxAPIC" rev 0x22 at pci0 dev 19 function 0 not configured
"Intel X58 Misc" rev 0x22 at pci0 dev 20 function 0 not configured
"Intel X58 GPIO" rev 0x22 at pci0 dev 20 function 1 not configured
"Intel X58 RAS" rev 0x22 at pci0 dev 20 function 2 not configured
"Intel X58 Throttle" rev 0x22 at pci0 dev 20 function 3 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 0 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 1 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 2 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 3 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 4 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 5 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 6 not configured
"Intel X58 QuickData" rev 0x22 at pci0 dev 22 function 7 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801JI USB" rev 0x00: apic 1 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801JI USB" rev 0x00: apic 1 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801JI USB" rev 0x00: apic 1 int 19
ehci0 at pci0 dev 26 function 7 "Intel 82801JI USB" rev 0x00: apic 1 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb4 at pci0 dev 28 function 0 "Intel 82801JI PCIE" rev 0x00: msi
pci5 at ppb4 bus 5
em0 at pci5 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
00:25:90:a4:e0:08
em1 at pci5 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
00:25:90:a4:e0:09
ppb5 at pci0 dev 28 function 4 "Intel 82801JI PCIE" rev 0x00: msi
pci6 at ppb5 bus 6
em2 at pci6 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
00:25:90:a4:e0:0a
em3 at pci6 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
00:25:90:a4:e0:0b
uhci3 at pci0 dev 29 function 0 "Intel 82801JI USB" rev 0x00: apic 1 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801JI USB" rev 0x00: apic 1 int 19
uhci5 at pci0 dev 29 function 2 "Intel 82801JI USB" rev 0x00: apic 1 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801JI USB" rev 0x00: apic 1 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb6 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x90
pci7 at ppb6 bus 7
vga1 at pci7 dev 1 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801JIR LPC" rev 0x00
ahci0 at pci0 dev 31 function 2 "Intel 82801JI AHCI" rev 0x00: msi, AHCI 1.2
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, INTEL SSDSC2CW06, 400i> SCSI3 0/direct
fixed naa.55cd2e40000c41e1
sd0: 57241MB, 512 bytes/sector, 117231408 sectors, thin
"Intel 82801JI SMBus" rev 0x00 at pci0 dev 31 function 3 not configured
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627DHG-P rev 0x73
lm1 at wbsio0 port 0xa10/8: W83627DHG
mtrr: Pentium Pro MTRR support
umass0 at uhub1 port 5 configuration 1 interface 0 "Genesys Logic USB Storage"
rev 2.00/0.16 addr 2
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets, initiator 0
cd0 at scsibus1 targ 1 lun 0: <TEAC, DV-28S-Y, 1.4A> SCSI0 5/cdrom removable
serial.05e30719000000000033
uhidev0 at uhub3 port 1 configuration 1 interface 0 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 variable keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (8dba4b197eb8a43b.a) swap on sd0b dump on sd0b
--
I'm not entirely sure you are real.
