Hello
I got one issue, and seems i need help from driver developers.
I have a some servers located in a different datacenters around
Russia, we used mostly 82575/827576 intel nic managed by e1000e and
igb drivers. When i testing speed with iperf from one 82576 card to
another - all working good, i can get 400-500-800mbit/s with one tcp
stream.
In last 5-6 months we bought some 10G intel 82599 cards and i see some
really strange issue, speed of tcp stream via 10G port is much slower
(from 3 to 10 times) than the 1G 82575/82576 link.
I have same server, same OS, same kernel - just difference is eth1 a
82575 (e1000e), and eth2 a 82599.
ethtool -i eth1
driver: ixgbe
version: 3.17.3
firmware-version: 0x61c10001
bus-info: 0000:03:00.0
ethtool -i eth2
driver: e1000e
version: 2.1.4-k
firmware-version: 1.8-0
bus-info: 0000:07:00.0
i bind different ip to each port
eth1 Link encap:Ethernet HWaddr 90:e2:ba:40:89:24
inet addr:xxx.xxx.185.135 Bcast:xxx.xxx.185.143 Mask:255.255.255.240
eth2 Link encap:Ethernet HWaddr 00:25:90:09:68:1b
inet addr:xxx.xxx.185.133 Bcast:xxx.xxx.185.143 Mask:255.255.255.240
and tried with iperf
this is 82575 fiber
------------------------------------------------------------
Client connecting to yyy.yyy.74.11, TCP port 5001
Binding to local address xxx.xxx.185.133
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.185.133 port 5001 connected with yyy.yyy.74.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-20.0 sec 1.03 GBytes 443 Mbits/sec
and this is 82599 optic
------------------------------------------------------------
Client connecting to yyy.yyy.74.11, TCP port 5001
Binding to local address xxx.xxx.185.135
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local xxx.xxx.185.135 port 5001 connected with yyy.yyy.74.11 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-20.1 sec 363 MBytes 151 Mbits/sec
The difference is impressive, cooper 82575 with e1000e done 3 times
faster than optical 82599.
Looks like ixgbe driver work in different way with tcp_window or maybe
tcp_congestion, because in realtime monitoring on 82575 i can see how
speed raise every second from 10MB/s to 90-100MB/s with 5-8MB/s step.
But on 82599 it just flowing around 10MB/s and won't go up.
i used this tuning to reach up to 900mbit
net.ipv4.tcp_rmem = 4096 87530 33554432
net.ipv4.tcp_wmem = 4096 65535 33554432
net.core.netdev_max_backlog = 3000
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_reordering = 100
net.ipv4.tcp_dsack = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 0
I use
Linux 3.8.0-26-generic #38-Ubuntu SMP Mon Jun 17 21:43:33 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 13.04
I ready to give any additional information and do any test.
I tried last driver from SF, turn off and on all offload, even tried
turn off interrupt moderation and set ring buffer to 64 to serve
traffic faster.
Both cards attached to same switch with 10G uplink.
--
Alexey Stoyanov
Kiev,Ukraine
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
