On Jan 18, 2011, at 6:51 AM, Claudio Jeker wrote:
> 165kpps is fairly low. Please add a dmesg so there is a chance to see what
> is causing this low rate. Modern HW with good nics should handle around
> 500kpps.
Good to know. Right now we're only on a 45Mbps connection at about 5kpps, so
that seemed fine to me. =) I've attached the dmesg below. It's nothing
fancy, as our needs aren't too huge. I was just pushing this particular
machine to its limits for fun to see what it could do since I had the traffic
generator on-site for a little extra time.
I didn't have a lot of time, so proper repeated testing wasn't really
possible. Even though we don't have the box anymore, I may set up some iperf
machines later to push it again just to verify the config (or test any
suggestions I receive from the list).
When the packet rate leveled off, we were at about 90% interrupt / 7% idle on
one CPU. The other CPU was (as expected), idle. Booting to SP didn't make a
noticeable difference. There were a significant number of ifq drops, though I
figured that was because the box was getting twice as much traffic in as it
was passing back out (and increasing maxlen substantially didn't do much).
The ruleset is simple:
=======================
table <ixiablue> { 192.168.100.101 } counters
table <ixiared> { 192.168.101.101 } counters
altq on trunk1 hfsc bandwidth 2Gb queue { other , toblue , tored }
queue other hfsc(realtime 10Mb linkshare 10Mb upperlimit 10Mb default)
queue toblue hfsc(realtime 100Mb linkshare 100Mb upperlimit 2Gb) qlimit 50
queue tored hfsc(realtime 100Mb linkshare 100Mb upperlimit 2Gb) qlimit 50
block log (all)
pass flags any
pass out on trunk1 from <ixiablue> to <ixiared> queue tored
pass out on trunk1 from <ixiared> to <ixiablue> queue toblue
=======================
Turning off queuing improved performance somewhat, but isn't helpful as a
real-world test since we rely on queuing in our production environment.
> Btw. trunk is using src & dest MAC addrs, a possible vlan tag and the IP /
> IPv6 src & dst addrs to build the hash. It does not use port numbers.
> The function used for this is trunk_hashmbuf().
Interesting. We definitely were seeing one of the interfaces in the trunk
totally unused for traffic outbound from the OpenBSD box. Since it doesn't use
port numbers, I suppose it's possible that we were just unlucky with the
src/dst IPs (we were only testing with two endpoint addresses and relying on
the multiple layer 4 ports to hash differently).
The trunk interface had a single IP (no vlans), and we were routing traffic to
it from our core switch using a policy routing scheme (we normally have a box
like this deployed in an off-path routing setup, so our core switch passes
traffic to it to be shaped). Not sure if a routed setup like that changes the
hash (it doesn't sound like it should), but I thought I'd mention it.
We can try again with more IP addresses next time to see if that spreads the
traffic a little more.
Thanks,
Jason
# dmesg
OpenBSD 4.8 (GENERIC.MP) #335: Mon Aug 16 09:09:20 MDT 2010
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 3486973952 (3325MB)
avail mem = 3380334592 (3223MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcfedf000 (39 entries)
bios0: vendor Phoenix Technologies LTD version "1.3a" date 11/03/2009
bios0: Supermicro X7SBi
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP _MAR MCFG HPET APIC BOOT SPCR ERST HEST BERT EINJ SLIC
SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT
acpi0: wakeup devices PXHA(S5) PEX_(S5) LAN_(S5) USB4(S5) USB5(S5) USB7(S5)
ESB2(S5) EXP1(S5) EXP5(S5) EXP6(S5) USB1(S5) USB2(S5) USB3(S5) USB6(S5)
ESB1(S5) PCIB(S5) KBC0(S1) MSE0(S1) COM1(S5) COM2(S5) PWRB(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3110 @ 3.00GHz, 3000.62 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,S
SSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG
cpu0: 6MB 64b/line 16-way L2 cache
cpu0: apic clock running at 333MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3110 @ 3.00GHz, 3000.21 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,S
SSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG
cpu1: 6MB 64b/line 16-way L2 cache
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 3 pa 0xfecc0000, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (PXHA)
acpiprt2 at acpi0: bus 3 (PEX_)
acpiprt3 at acpi0: bus 5 (EXP1)
acpiprt4 at acpi0: bus 13 (EXP5)
acpiprt5 at acpi0: bus 15 (EXP6)
acpiprt6 at acpi0: bus 17 (PCIB)
acpicpu0 at acpi0: C3, PSS
acpicpu1 at acpi0: C3, PSS
acpibtn0 at acpi0: PWRB
acpivideo0 at acpi0: IGD0
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 3000 MHz: speeds: 3000, 2667, 2333, 2000 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: apic 2 int 16
(irq 5)
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci2 at ppb1 bus 2
"Intel IOxAPIC" rev 0x09 at pci1 dev 0 function 1 not configured
ppb2 at pci0 dev 6 function 0 "Intel 3210 PCIE" rev 0x01: apic 2 int 16 (irq
5)
pci3 at ppb2 bus 3
em0 at pci3 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2
int 16 (irq 5), address 00:15:17:d6:6a:3c
em1 at pci3 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic 2
int 17 (irq 10), address 00:15:17:d6:6a:3d
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 16
(irq 5)
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 17
(irq 10)
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 18
(irq 11)
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 18
(irq 11)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 2 int 16
(irq 5)
pci4 at ppb3 bus 5
ppb4 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 2 int 16
(irq 5)
pci5 at ppb4 bus 13
em2 at pci5 dev 0 function 0 "Intel PRO/1000MT (82573E)" rev 0x03: apic 2 int
16 (irq 5), address 00:30:48:fb:3d:76
ppb5 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: apic 2 int 17
(irq 10)
pci6 at ppb5 bus 15
em3 at pci6 dev 0 function 0 "Intel PRO/1000MT (82573L)" rev 0x00: apic 2 int
17 (irq 10), address 00:30:48:fb:3d:77
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 23
(irq 10)
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 22
(irq 11)
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 18
(irq 11)
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 23
(irq 10)
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb6 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci7 at ppb6 bus 17
vga1 at pci7 dev 3 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 2 int 22 (irq 11)
drm0 at radeondrm0
pciide0 at pci7 dev 4 function 0 "ITExpress IT8213F" rev 0x00: DMA
(unsupported), channel 0 wired to native-PCI, channel 1 wired to native-PCI
pciide0: using apic 2 int 23 (irq 10) for native-PCI interrupt
pciide0: channel 0 ignored (not responding; disabled or no drives?)
pciide0: channel 1 ignored (not responding; disabled or no drives?)
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: apic 2 int 17
(irq 10), AHCI 1.2
scsibus0 at ahci0: 32 targets
cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, CDDVDW SN-S083B, SB02> ATAPI 5/cdrom
removable
sd0 at scsibus0 targ 1 lun 0: <ATA, INTEL SSDSA2M040, 2CV1> SCSI3 0/direct
fixed
sd0: 38166MB, 512 bytes/sec, 78165360 sec total
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 2 int 17
(irq 10)
iic0 at ichiic0
lm1 at iic0 addr 0x2d: W83627HF
wbng0 at iic0 addr 0x2f: w83793g
spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM ECC PC2-6400CL5
spdmem1 at iic0 addr 0x52: 2GB DDR2 SDRAM ECC PC2-6400CL5
"Intel 82801I Thermal" rev 0x02 at pci0 dev 31 function 6 not configured
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627HF rev 0x41
lm2 at wbsio0 port 0x290/8: W83627HF
lm1 detached
mtrr: Pentium Pro MTRR support
uhub8 at uhub5 port 2 "Mitsumi Electric Hub in Apple USB Keyboard" rev
1.10/2.11 addr 2
uhidev0 at uhub8 port 1 configuration 1 interface 0 "Mitsumi Electric Apple
USB Keyboard" rev 1.00/1.03 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub8 port 2 configuration 1 interface 0 "MacALLY MacALLY USB One
Button Mouse" rev 1.00/2.00 addr 4
uhidev1: iclass 3/1
ums0 at uhidev1: 6 buttons, Z dir
wsmouse0 at ums0 mux 0
softraid0 at root
root on sd0a swap on sd0b dump on sd0b
ichiic0: abort failed, status 0x40<INUSE>
ichiic0: abort failed, status 0x0
