root wrote:
Summary: Using iperf to measure TCP net speed between a linux and
freebsd box over gigE, I see significant speed difference depending on
the data direction. Pushing data from the freebsd box to the linux box,
I average about 500Gb/s. Pushing data from the linux box to the freebsd
box, I see about 300Gb/s. It would seem that there is something
limiting either the transmit side of the linux box or the recv side of
the freebsd box. Any suggestions on how to narrow down who's tuning
needs more work would also be welcome. Just for grins, I tried upping
the txqueuelen on the linux box to 1500, but no help.
I'd like to get closer to the theoretical speed of this link (I'd be
happy if I could get 600-700 both ways), but the real puzzler is the
difference. The client is the linux box @192.168.1.104 and the iperf
server is the freebsd box @ 192.168.1.15 . Neither box is cpu or net
busy during testing. Both systems only have one nic. I played with
larger window sizes, but it only had a minor affect. I went with the
Intel Pro/1000's based on their good rep in both the linux and freebsd
communities.
Client connecting to 192.168.1.15, TCP port 5001
TCP window size: 128 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001
[ 6] 0.0-10.0 sec 369 MBytes 309 Mbits/sec
[ 5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963
[ 5] 0.0-10.0 sec 597 MBytes 500 Mbits/sec
Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are
Belkin 5e), both systems are on this switch.
FreeBSD box:
FreeBSD 6.1-RELEASE-p10
1.9Ghz Athlon / 1 gig of main mem
Abit/nforce2 MB+chipset (onboard nic disabled in bios)
Intel Pro/1000GT NIC
sysctl.conf:
kern.ipc.maxsockbuf=8192000
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
sysctl reports:
kern.ipc.nmbjumbo16: 0
kern.ipc.nmbjumbo9: 0
kern.ipc.nmbjumbop: 0
kern.ipc.nmbclusters: 25600
net.inet.tcp.rfc1323: 1
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_MTU>
inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1
inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255
ether 00:0e:0c:da:1a:3c
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
Linux box:
Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686
GNU/Linux
Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem.
I run tcp_tune.sh at boot on the linux box
#!/bin/bash
echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem
echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem
echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem
/proc/sys/net/ipv4/tcp_window_scaling is 1
/proc/sys/net/ipv4/tcp_timestamps is 1
eth1 Link encap:Ethernet HWaddr 00:0E:0C:DA:1A:3B
inet addr:192.168.1.104 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0
TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:147638720 (140.7 MiB) TX bytes:1109677958 (1.0 GiB)
Base address:0xd800 Memory:e2020000-e2040000
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
Link detected: yes
Have you tried upping the MTU, that is if the cards and switch you are
using support it. I have seen significant speed increases (FreeBSD -
FreeBSD) in some scenario's twice the through put with an MTU of 8192.
I have also seen 700Mb/s sustained FreeBSD - FreeBSD using the openssh
HPN patch set and no extra tuning of the network stack. Which makes me
think that maybe the linux stack needs some tuning?
Just some ideas
Tom
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
