Have you tried the multi-link? Did it helped? George.
On Apr 21, 2014, at 10:34 , Muhammad Ansar Javed <muhammad.an...@seecs.edu.pk> wrote: > I am able to achieve around 90% ( maximum 9390 Mbps) bandwidth on 10GE. There > were configuration issues disabling Intel Speedstep and Interrupt coalescing > helped in achieving expected network bandwidth. Varying send and recv buffer > sizes from 128 KB to 1 MB added just 50 Mbps with maximum bandwidth achieved > on 1 MB buffer size. > Thanks for support. > > > On Thu, Apr 17, 2014 at 6:05 AM, George Bosilca <bosi...@icl.utk.edu> wrote: > Muhammad, > > Our configuration of TCP is tailored for 1Gbs networks, so it’s performance > on 10G might be sub-optimal. That being said, the remaining of this email > will be speculation as I do not have access to a 10G system to test it. > > There are two things that I would test to see if I can improve the > performance. > > 1. The send and receive TCP suffer. These are handled by the btl_tcp_sndbuf > and btl_tcp_rcvbuf. By default these are set to 128K which is extremely small > for a 10G network. Try 256KB or maybe even 1M (you might need to fiddle with > your kernel to get here). > > 2. Add more links between the processes by increasing the default value for > btl_tcp_links to 2 or 4. > > You might also try to the following (but here I’m more skeptical). Try > pushing the value of btl_tcp_endpoint_cache up. This parameter is not to be > used eagerly in real applications with a complete communication pattern, but > for a benchmark it might be a good use. > > George. > > On Apr 16, 2014, at 06:30 , Muhammad Ansar Javed > <muhammad.an...@seecs.edu.pk> wrote: > >> Hi Ralph, >> Yes, you are right. I should have also tested NetPipe-MPI version earlier. I >> ran NetPipe-MPI version on 10G Ethernet and maximum bandwidth achieved is >> 5872 Mbps. Moreover, maximum bandwidth achieved by osu_bw test is 6080 Mbps. >> I have used OSU-Micro-Benchmarks version 4.3. >> >> >> On Wed, Apr 16, 2014 at 3:40 PM, Ralph Castain <r...@open-mpi.org> wrote: >> I apologize, but I am now confused. Let me see if I can translate: >> >> * you ran the non-MPI version of the NetPipe benchmark and got 9.5Gps on a >> 10Gps network >> >> * you ran iperf and got 9.61Gps - however, this has nothing to do with MPI. >> Just tests your TCP stack >> >> * you tested your bandwidth program on a 1Gps network and got about 90% >> efficiency. >> >> Is the above correct? If so, my actual suggestion was to run the MPI version >> of NetPipe and to use the OSB benchmark program as well. Your program might >> well be okay, but benchmarking is a hard thing to get right in a parallel >> world, so you might as well validate it by cross-checking the result. >> >> I suggest this mostly because your performance numbers are far worse than >> anything we've measured using those standard benchmarks, and so we should >> first ensure we aren't chasing a ghost. >> >> >> >> >> >> On Wed, Apr 16, 2014 at 1:41 AM, Muhammad Ansar Javed >> <muhammad.an...@seecs.edu.pk> wrote: >> Yes, I have tried NetPipe-Java and iperf for bandwidth and configuration >> test. NetPipe Java achieves maximum 9.40 Gbps while iperf achieves maximum >> 9.61 Gbps bandwidth. I have also tested my bandwidth program on 1Gbps >> Ethernet connection and it achieves 901 Mbps bandwidth. I am using the same >> program for 10G network benchmarks. Please find attached source file of >> bandwidth program. >> >> As far as --bind-to core is concerned, I think it is working fine. Here is >> output of --report-bindings switch. >> [host3:07134] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././.] >> [host4:10282] MCW rank 1 bound to socket 0[core 0[hwt 0]]: [B/././.] >> >> >> >> >> On Tue, Apr 15, 2014 at 8:39 PM, Ralph Castain <r...@open-mpi.org> wrote: >> Have you tried a typical benchmark (e.g., NetPipe or OMB) to ensure the >> problem isn't in your program? Outside of that, you might want to explicitly >> tell it to --bind-to core just to be sure it does so - it's supposed to do >> that by default, but might as well be sure. You can check by adding >> --report-binding to the cmd line. >> >> >> On Apr 14, 2014, at 11:10 PM, Muhammad Ansar Javed >> <muhammad.an...@seecs.edu.pk> wrote: >> >>> Hi, >>> I am trying to benchmark Open MPI performance on 10G Ethernet network >>> between two hosts. The performance numbers of benchmarks are less than >>> expected. The maximum bandwidth achieved by OMPI-C is 5678 Mbps and I was >>> expecting around 9000+ Mbps. Moreover latency is also quite higher than >>> expected, ranging from 37 to 59 us. Here is complete set of numbers. >>> >>> Latency >>> Open MPI C >>> Size Time (us) >>> 1 37.76 >>> 2 37.75 >>> 4 37.78 >>> 8 55.17 >>> 16 37.89 >>> 32 39.08 >>> 64 37.78 >>> 128 59.46 >>> 256 39.37 >>> 512 40.39 >>> 1024 47.18 >>> 2048 47.84 >>> >>> >>> Bandwidth >>> Open MPI C >>> Size (Bytes) Bandwidth (Mbps) >>> 2048 412.22 >>> 4096 539.59 >>> 8192 827.73 >>> 16384 1655.35 >>> 32768 3274.3 >>> 65536 1995.22 >>> 131072 3270.84 >>> 262144 4316.22 >>> 524288 5019.46 >>> 1048576 5236.17 >>> 2097152 5362.61 >>> 4194304 5495.2 >>> 8388608 5565.32 >>> 16777216 5678.32 >>> >>> >>> My environments consists of two hosts having point-to-point (switch-less) >>> 10Gbps Ethernet connection. Environment (OS, user, directory structure >>> etc) on both hosts is exactly same. There is no NAS or shared file system >>> between both hosts. Following are configuration and job launching commands >>> that I am using. Moreover, I have attached output of script ompi_info --all. >>> >>> Configuration commmand: ./configure --enable-mpi-java >>> --prefix=/home/mpj/installed/openmpi_installed CC=/usr/bin/gcc >>> --disable-mpi-fortran >>> >>> Job launching command: mpirun -np 2 -hostfile machines -npernode 1 >>> ./latency.out >>> >>> Are these numbers okay? If not then please suggest performance tuning >>> steps... >>> >>> Thanks >>> >>> -- >>> Ansar Javed >>> HPC Lab >>> SEECS NUST >>> Contact: +92 334 438 9394 >>> Email: muhammad.an...@seecs.edu.pk >>> <ompi_info.tar.bz2>_______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> >> -- >> Regards >> >> >> Ansar Javed >> HPC Lab >> SEECS NUST >> Contact: +92 334 438 9394 >> Email: muhammad.an...@seecs.edu.pk >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> >> -- >> Regards >> >> Ansar Javed >> HPC Lab >> SEECS NUST >> Contact: +92 334 438 9394 >> Email: muhammad.an...@seecs.edu.pk >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > -- > Regards > > Ansar Javed > HPC Lab > SEECS NUST > Contact: +92 334 438 9394 > Email: muhammad.an...@seecs.edu.pk > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
