It sounds like what you’re asking is “how do I get the best performance from Open MPI in AWS?”.
The TCP BTL is your best option for performance in AWS. RoCE is going to be a bunch of work to get setup, and you’ll still end up with host processing of every packet. There are a couple simple instance tweaks that can make a big difference. AWS has published a very nice guide for setting up an EDA workload environment [1], which has a number of useful tweaks, particularly if you’re using C4 or earlier compute instances. The biggest improvement, however, is to make sure you’re using a version of Open MPI newer than 2.1.2. We fixed some fairly serious performance issues in the Open MPI TCP stack (that, humorously enough, were also in the MPICH TCP stack and have been fixed there as well) in 2.1.2. Given that your application is fairly asynchronous, you might want to experiment with the btl_tcp_progress_thread MCA parameter. If your application benefits from asynchronous progress, using a progress thread might be the best option. Brian > On Sep 6, 2018, at 7:10 PM, Benjamin Brock <br...@cs.berkeley.edu> wrote: > > I'm setting up a cluster on AWS, which will have a 10Gb/s or 25Gb/s Ethernet > network. Should I expect to be able to get RoCE to work in Open MPI on AWS? > > More generally, what optimizations and performance tuning can I do to an Open > MPI installation to get good performance on an Ethernet network? > > My codes use a lot of random access AMOs and asynchronous block transfers, so > it seems to me like setting up RDMA over Ethernet would be essential to > getting good performance, but I can't seem to find much information about it > online. > > Any pointers you have would be appreciated. > > Ben > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
