Robert Watson wrote:
On Fri, 14 Oct 2005, Michael VInce wrote:
I been doing some network benchmarking using netperf and just simple
'fetch' on a new network setup to make sure I am getting the most out
of the router and servers, I thought I would post some results in
case some one can help me with my problems or if others are just
interested to see the results.
Until recently (or maybe still), netperf was compiled with -DHISTOGRAM
by our port/package, which resulted in a significant performance
drop. I believe that the port maintainer and others have agreed to
change it, but I'm not sure if it's been committed yet, or which
packages have been rebuilt. You may want to manually rebuild it to
make sure -DHISTOGRAM isn't set.
You may want to try setting net.isr.direct=1 and see what performance
impact that has for you.
Robert N M Watson
I reinstalled the netperf to make sure its the latest.
I have also decided to upgrade Server-C (the i386 5.4 box) to 6.0RC1 and
noticed it gave a large improvement of network performance with a SMP
kernel.
As with the network setup ( A --- B --- C ) with server B being the
gateway, doing a basic 'fetch' from the gateway (B) to the Apache server
(C) it gives up to 700mbits/sec transfer performance, doing a fetch from
server A thus going through the gateway gives slower but still decent
performance of up to 400mbits/sec.
B> fetch -o - > /dev/null http://server-c/file1gig.iso
- 100% of 1055 MB 69 MBps
00m00s
A> fetch -o - > /dev/null http://server-c/file1gig.iso
- 100% of 1055 MB 39 MBps
00m00s
Netperf from the gateway directly to the apache server (C) 916mbits/sec
B> /usr/local/netperf/netperf -l 20 -H server-C -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
Elapsed Throughput - 10^6bits/sec: 916.50
Netperf from the client machine through the gateway to the apache server
(C) 315mbits/sec
A> /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
Elapsed Throughput - 10^6bits/sec: 315.89
Client to gateway netperf test shows the direct connection between these
machines is fast. 912mbits/sec
A> /usr/local/netperf/netperf -l 30 -H server-B -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 5734
Elapsed Throughput - 10^6bits/sec: 912.11
The strange thing now is in my last post I was able to get faster speeds
from server A to C with 'fetch' tests on non-smp kernels and slower
speeds with netperf tests. Now I get speeds a bit slower with fetch
tests but faster netperf speed tests with or without SMP on server-C.
I was going to test with 'net.isr.dispatch' but the sysctl doesn't
appear to exist, doing this returns nothing.
'sysctl -a | grep 'net.isr.dispatch'
I also tried polling but its also like that doesn't exist either.
ifconfig em3 inet 192.168.1.1 netmask 255.255.255.224 polling
ifconfig: polling: Invalid argument
When doing netperf tests there was high interrupt usage.
CPU states: 0.7% user, 0.0% nice, 13.5% system, 70.0% interrupt, 15.7%
idle
Also the server B is using its last 2 gigabit ethernet ports which are
listed from pciconf -lv as '82547EI Gigabit Ethernet Controller'
While the first 2 are listed as 'PRO/1000 P'
Does any one know if the PRO/1000P would be better?
[EMAIL PROTECTED]:4:0: class=0x020000 card=0x118a8086 chip=0x108a8086 rev=0x03
hdr=0x00
vendor = 'Intel Corporation'
device = 'PRO/1000 P'
[EMAIL PROTECTED]:8:0: class=0x020000 card=0x016d1028 chip=0x10768086 rev=0x05
hdr=0x00
vendor = 'Intel Corporation'
device = '82547EI Gigabit Ethernet Controller'
Cheers,
Mike
The network is currently like this, where machines A and B are the
Dell 1850s and C is the 2850 x 2 CPU (Server C has Apache2 worker MPM
on it) and server B is the gateway and A is acting as a client for
fetch and netperf tests.
A --- B --- C
The 2 1850s are running AMD64 Freebsd 6.0rc1 (A and B) while C is
running 5.4-stable i386 from Oct 12
My main problem is that if I compile SMP into the machine C
(5.4stable) the network speed goes down to a range between
6mbytes/sec to 15mbytes/sec on SMP.
If I use GENERIC kernel the performance goes up to what I have show
below which is around 65megabytes/sec for a 'fetch' get test from
Apache server and 933mbits/sec for netperf.
Does any know why why network performance would be so bad on SMP?
Does any one think that if I upgrade the i386 SMP server to 6.0RC1
the SMP network performance would improve? This server will be
running java so I need it to be stable and is the the reason I am
using i386 and Java 1.4
I am happy with performance of direct machine to machine (non SMP)
which is pretty much full 1gigabit/sec speeds.
Going through the gateway server-B seems to drop its speed down a bit
for in and out direction tcp speed tests using netperf I get around
266mbits/sec from server A through gateway Server-B to server-C which
is quite adequate for the link I currently have for it.
Doing a 'fetch' get for a 1gig file from the Apache server gives good
speeds of close to 600mbits/sec but netperf shows its weakness with
266mbits/sec.
This is as fast as I need it to be but does any one know the weak
points on the router gateway to make it faster? Is this the
performance I should expect for FreeBSD as a router with gigabit ethers?
I have seen 'net.inet.ip.fastforwarding' in some peoples router
setups on the list but nothing about what it does or what it can affect.
I haven't done any testing with polling yet but if I can get over
900mbits/sec on the interfaces does polling help with passing packets
from one interface to the other?
All machines have PF running other then that they don't really have
any sysctls or special kernel options.
Here are some speed benchmarks using netperf and 'fetch' gets.
Server A to server C with server C using SMP kernel and just GENERIC
kernel further below
B# /usr/local/netperf/netperf -l 10 -H server-C -t TCP_STREAM -i 10,2
-I 99,5 -- -m 4096 -s 57344 -S 57344
TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
57344 57344 4096 10.06 155.99
tank# fetch -o - > /dev/null http://server-C/file1gig.iso
- 100% of 1055 MB 13
MBps 00m00s
##### Using generic non SMP kernel
Server A to server C with server C using GENERIC kernel.
A# fetch -o - > /dev/null http://server-C/file1gig.iso
- 100% of 1055 MB 59
MBps 00m00s
A# ./tcp_stream_script server-C
/usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
57344 57344 4096 60.43 266.92
------------------------------------
###############################################
Connecting from server-A to B (gateway)
A# ./tcp_stream_script server-B
------------------------------------
/usr/local/netperf/netperf -l 60 -H server-B -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
TCP STREAM TEST to server-B : +/-2.5% @ 99% conf. : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
57344 57344 4096 61.80 926.82
------------------------------------
##########################################
Connecting from server B (gateway) to server C
Fetch and Apache2 test
B# fetch -o - > /dev/null http://server-C/file1gig.iso
- 100% of 1055 MB 74
MBps 00m00s
Netperf test
B# /usr/local/netperf/tcp_stream_script server-C
/usr/local/netperf/netperf -l 60 -H server-C -t TCP_STREAM -i 10,2 -I
99,5 -- -m 4096 -s 57344 -S 57344
TCP STREAM TEST to server-C : +/-2.5% @ 99% conf. : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
57344 57344 4096 62.20 933.94
------------------------------------
Cheers,
Mike
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
_______________________________________________
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
