I switched over to X1 instances in AWS which have 20 Gbps connectivity. Using iperf3, I'm seeing 11.1 Gbps between them with just one port. iperf3 supports a -P option which will connect using multiple ports... Setting this to use in the range of 5-20 ports (there's some variability from run to run), I can get in the range of 18 Gbps aggregate which for a real world speed seems pretty good.
Using mpirun with the previously-suggested btl_tcp_sndbuf and btl_tcp_rcvbuf settings, I'm getting around 10.7 Gbps. So, pretty close to iperf with just one port (makes sense there'd be some overhead with MPI). My understanding of the btl_tcp_links flag that Gilles mentioned is that it should be analogous to iperf's -P flag - it should connect with multiple ports in the hopes of improving the aggregate bandwidth. If that's what this flag is supposed to do, it does not appear to be working properly for me. With lsof, I can see the expected number of ports show up when I run iperf. However, with MPI I only ever see three connections between the two machines - sshd, orted, and my actual application. No matter what I set btl_tcp_links to, I don't see any additional ports show up (or any change in performance). Am I misunderstanding what this flag does or is there a bug here? If I am misunderstanding the flag's intent, is there a different flag that would allow Open MPI to use multiple ports similar to what iperf is doing? Thanks. -Adam On Mon, Jul 10, 2017 at 9:31 PM, Adam Sylvester <op8...@gmail.com> wrote: > Thanks again Gilles. Ahh, better yet - I wasn't familiar with the config > file way to set these parameters... it'll be easy to bake this into my AMI > so that I don't have to set them each time while waiting for the next Open > MPI release. > > Out of mostly laziness I try to keep to the formal releases rather than > applying patches myself, but thanks for the link to it (the commit comments > were useful to understand why this improved performance). > > -Adam > > On Mon, Jul 10, 2017 at 12:04 AM, Gilles Gouaillardet <gil...@rist.or.jp> > wrote: > >> Adam, >> >> >> Thanks for letting us know your performance issue has been resolved. >> >> >> yes, https://www.open-mpi.org/faq/?category=tcp is the best place to >> look for this kind of information. >> >> i will add a reference to these parameters. i will also ask folks at AWS >> if they have additional/other recommendations. >> >> >> note you have a few options before 2.1.2 (or 3.0.0) is released : >> >> >> - update your system wide config file (/.../etc/openmpi-mca-params.conf) >> or user config file >> >> ($HOME/.openmpi/mca-params.conf) and add the following lines >> >> btl_tcp_sndbuf = 0 >> >> btl_tcp_rcvbuf = 0 >> >> >> - add the following environment variable to your environment >> >> export OMPI_MCA_btl_tcp_sndbuf=0 >> >> export OMPI_MCA_btl_tcp_rcvbuf=0 >> >> >> - use Open MPI 2.0.3 >> >> >> - last but not least, you can manually download and apply the patch >> available at >> >> https://github.com/open-mpi/ompi/commit/b64fedf4f652cadc9bfc >> 7c4693f9c1ef01dfb69f.patch >> >> >> Cheers, >> >> Gilles >> >> On 7/9/2017 11:04 PM, Adam Sylvester wrote: >> >>> Gilles, >>> >>> Thanks for the fast response! >>> >>> The --mca btl_tcp_sndbuf 0 --mca btl_tcp_rcvbuf 0 flags you recommended >>> made a huge difference - this got me up to 5.7 Gb/s! I wasn't aware of >>> these flags... with a little Googling, is https://www.open-mpi.org/faq/? >>> category=tcp the best place to look for this kind of information and >>> any other tweaks I may want to try (or if there's a better FAQ out there, >>> please let me know)? >>> There is only eth0 on my machines so nothing to tweak there (though good >>> to know for the future). I also didn't see any improvement by specifying >>> more sockets per instance. But, your initial suggestion had a major impact. >>> In general I try to stay relatively up to date with my Open MPI version; >>> I'll be extra motivated to upgrade to 2.1.2 so that I don't have to >>> remember to set these --mca flags on the command line. :o) >>> -Adam >>> >>> On Sun, Jul 9, 2017 at 9:26 AM, Gilles Gouaillardet < >>> gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> >>> wrote: >>> >>> Adam, >>> >>> at first, you need to change the default send and receive socket >>> buffers : >>> mpirun --mca btl_tcp_sndbuf 0 --mca btl_tcp_rcvbuf 0 ... >>> /* note this will be the default from Open MPI 2.1.2 */ >>> >>> hopefully, that will be enough to greatly improve the bandwidth for >>> large messages. >>> >>> >>> generally speaking, i recommend you use the latest (e.g. Open MPI >>> 2.1.1) available version >>> >>> how many interfaces can be used to communicate between hosts ? >>> if there is more than one (for example a slow and a fast one), you'd >>> rather only use the fast one. >>> for example, if eth0 is the fast interface, that can be achieved with >>> mpirun --mca btl_tcp_if_include eth0 ... >>> >>> also, you might be able to achieve better results by using more than >>> one socket on the fast interface. >>> for example, if you want to use 4 sockets per interface >>> mpirun --mca btl_tcp_links 4 ... >>> >>> >>> >>> Cheers, >>> >>> Gilles >>> >>> On Sun, Jul 9, 2017 at 10:10 PM, Adam Sylvester <op8...@gmail.com >>> <mailto:op8...@gmail.com>> wrote: >>> > I am using Open MPI 2.1.0 on RHEL 7. My application has one >>> unavoidable >>> > pinch point where a large amount of data needs to be transferred >>> (about 8 GB >>> > of data needs to be both sent to and received all other ranks), >>> and I'm >>> > seeing worse performance than I would expect; this step has a >>> major impact >>> > on my overall runtime. In the real application, I am using >>> MPI_Alltoall() >>> > for this step, but for the purpose of a simple benchmark, I >>> simplified to >>> > simply do a single MPI_Send() / MPI_Recv() between two ranks of >>> a 2 GB >>> > buffer. >>> > >>> > I'm running this in AWS with instances that have 10 Gbps >>> connectivity in the >>> > same availability zone (according to tracepath, there are no >>> hops between >>> > them) and MTU set to 8801 bytes. Doing a non-MPI benchmark of >>> sending data >>> > directly over TCP between these two instances, I reliably get >>> around 4 Gbps. >>> > Between these same two instances with MPI_Send() / MPI_Recv(), I >>> reliably >>> > get around 2.4 Gbps. This seems like a major performance >>> degradation for a >>> > single MPI operation. >>> > >>> > I compiled Open MPI 2.1.0 with gcc 4.9.1 and default settings. I'm >>> > connecting between instances via ssh and using I assume TCP for >>> the actual >>> > network transfer (I'm not setting any special command-line or >>> programmatic >>> > settings). The actual command I'm running is: >>> > mpirun -N 1 --bind-to none --hostfile hosts.txt my_app >>> > >>> > Any advice on other things to test or compilation and/or runtime >>> flags to >>> > set would be much appreciated! >>> > -Adam >>> > >>> > _______________________________________________ >>> > users mailing list >>> > users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>> > https://rfd.newmexicoconsortium.org/mailman/listinfo/users >>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users> >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org> >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users> >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >>> >> >> _______________________________________________ >> users mailing list >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users >> > >
_______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
