Hi
I have bought two lab systems for testing 10Gbit and have some issues with
10Gbit performance.
Note: I have the devices triple-booting FreeBSD, Linux and OpenBSD and can get
about 9.4Gbits/sec with Linux iperf using iperf -t 60
I have read and updated kernel to latest snapshot
OpenBSD 5.2 (GENERIC.MP) #362: Sun Jul 22 14:10:40 MDT 2012
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17169907712 (16374MB)
avail mem = 16690479104 (15917MB)
also tried GENERIC-nonMP - gave worse performance
Current performance, a little better than 5.1
root@xpc02:root# tcpbench -u 10.0.10.30
Elapsed: 1000 Mbps: 1841.437 Peak Mbps: 1841.437 Tx PPS: 156372
Elapsed: 2000 Mbps: 1841.901 Peak Mbps: 1841.901 Tx PPS: 156411
Elapsed: 3000 Mbps: 1839.553 Peak Mbps: 1841.901 Tx PPS: 156212
root@xpc03:root# tcpbench -n 4 -s
elapsed_ms bytes mbps bwidth
999 92113072 737.642 27.54%
999 98731880 790.646 29.52%
999 72763448 582.690 21.75%
999 70892632 567.709 21.19%
Conn: 4 Mbps: 2678.687 Peak Mbps: 2678.687 Avg Mbps: 669.672
2000 65977576 527.821 18.87%
1999 107496584 860.834 30.75%
1999 67310912 539.026 19.25%
1999 108839944 871.591 31.13%
Conn: 4 Mbps: 2799.272 Peak Mbps: 2799.272 Avg Mbps: 699.818
OTOH it seems to be very stable, even running over longer periods :-)
Better stable than fast :-)
Basic specs
Shuttle SX58H7 Pro Barebone, using SX58 chipset - has x8 PCI-E slot
Dual port 10Gbit Intel 10GbE SFP+ (82599)
cpu0: Intel(R) Core(TM) i7 CPU 960 @ 3.20GHz, 3200.46 MHz
spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600
spdmem1 at iic0 addr 0x51: 4GB DDR3 SDRAM PC3-10600
spdmem2 at iic0 addr 0x52: 4GB DDR3 SDRAM PC3-10600
spdmem3 at iic0 addr 0x54: 4GB DDR3 SDRAM PC3-10600, BIOS overclocking memory
a bit up to 1333MHz
Current tuning, apart from kernel defaults:
and also modified /etc/sysctl somewhat - most changes below are from
https://calomel.org/network_performance.html:
hlk@xpc02:hlk$ grep -v "^#" /etc/sysctl.conf
net.inet6.icmp6.rediraccept=1 # 1=Accept IPv6 ICMP redirects (for hosts)
kern.pool_debug=0 # 0=Disable pool corruption checks (faster)
machdep.allowaperture=2 # See xf86(4)
ddb.panic=0 # do not enter ddb console on kernel panic,
reboot if possible
kern.bufcachepercent=90 # Allow the kernel to use up to 90% of the RAM
for cache (default 10%)
machdep.allowaperture=2 # Access the X Window System (if you use X on
the system)
net.inet.ip.forwarding=1 # Permit forwarding (routing) of packets
through the firewall
net.inet.ip.ifq.maxlen=512 # Maximum allowed input queue length
(256*number of physical interfaces)
net.inet.ip.mtudisc=0 # TCP MTU (Maximum Transmission Unit) discovery
off since our mss is small enough
net.inet.tcp.rfc3390=2 # RFC3390 increasing TCP's Initial Congestion
Window to 14600 for SPDY
net.inet.tcp.mssdflt=1440 # maximum segment size (1440 from scrub pf.conf
match statement)
net.inet.udp.recvspace=131072 # Increase UDP "receive" buffer size. Good for
200Mbit without packet drop.
net.inet.udp.sendspace=131072 # Increase UDP "send" buffer size. Good for
200Mbit without packet drop.
I have a more detailed pastebin at: http://pastebin.com/iNpk3DgV
The devices are NOT intended for production and I will be happy to allow
access to these systems, only requirement is that you have IPv6 (I am moving
these systems around a bit, and NAT sucks)
- email me your SSH public key and I will create an account.
I have this week until saturday dedicated for working on this and other
related projects, so contact me if you want to join. Skype: kramshoej, EFnet:
hlkv6, freenode: hlkv6
Current cabling is using direct attach copper cable, I only have direct attach
cables, but multiple switches, did not make any significant difference.
I got a Force10 switch to report the following in testing, so the hardware
looks fine:
force10-2#sh interfaces Tengigabitethernet 0/0 | grep Mbits
Input 35.00 Mbits/sec, 62616 packets/sec, 0.45% of line-rate
Output 9869.00 Mbits/sec, 812987 packets/sec, 99.90% of line-rate
The main steps for getting FreeBSD tuned were:
kern.ipc.nmbclusters=262144
kern.ipc.nmbjumbop=262144
sysctl hw.intr_storm_threshold=9000 - note I see LOTS and lots of interrupts,
so maybe some
/boot/loader.conf
ixgbe_load="YES"
hw.ixgbe.txd=4096
hw.ixgbe.rxd=4096
- this made a huge difference
Killing xhci from kernel, was using lots of interrupts, buggy driver?
Best regards
Henrik
--
Henrik Lund Kramshøj, Follower of the Great Way of Unix
h...@kramse.org h...@solidonetworks.com
+45 2026 6000 cand.scient CISSP
http://solidonetworks.com/ Network Security is a business enabler
