On 02/29/2012 03:15 PM, Jeffrey Squyres wrote: > On Feb 29, 2012, at 2:57 PM, Jingcha Joba wrote: > >> So if I understand correctly, if a message size is smaller than it will use >> the MPI way (non-RDMA, 2 way communication), if its larger, then it would >> use the Open Fabrics, by using the ibverbs (and ofed stack) instead of using >> the MPI's stack? > Er... no. > > So let's talk MPI-over-OpenFabrics-verbs specifically. > > All MPI communication calls will use verbs under the covers. They may use > verbs send/receive semantics in some cases, and RDMA semantics in other > cases. "It depends" -- on a lot of things, actually. It's hard to come up > with a good rule of thumb for when it uses one or the other; this is one of > the reasons that the openib BTL code is so complex. :-) > > The main points here are: > > 1. you can trust the openib BTL to do the Best thing possible to get the > message to the other side. Regardless of whether that message is an MPI_SEND > or an MPI_PUT (for example). > > 2. MPI_PUT does not necessarily == verbs RDMA write (and likewise, MPI_GET > does not necessarily == verbs RDMA read). > >> If so, could that be the reason why the MPI_Put "hangs" when sending a >> message more than 512KB (or may be 1MB)? > No. I'm guessing that there's some kind of bug in the MPI_PUT implementation. > >> Also is there a way to know if for a particular MPI call, OF uses send/recv >> or RDMA exchange? > Not really. > > More specifically: all things being equal, you don't care which is used. You > just want your message to get to the receiver/target as fast as possible. > One of the main ideas of MPI is to hide those kinds of details from the user. > I.e., you call MPI_SEND. A miracle occurs. The message is received on the > other side. > > :-) >
Nice use of the "A Miracle Occurs" meme. We really need t-shirts that say this for the OpenMPI BoF at SC12. -- Prentice
