On 10/22/13 10:23 AM, "Jai Dayal" <dayals...@gmail.com<mailto:dayals...@gmail.com>> wrote:
I, for the life of me, can't understand the difference between these two init_thread modes. MPI_THREAD_SINGLE states that "only one thread will execute", but MPI_THREAD_FUNNELED states "The process may be multi-threaded, but only the main thread will make MPI calls (all MPI calls are funneled to the main thread)." If I use MPI_THREAD_SINGLE, and just create a bunch of pthreads that dumbly loop in the background, the MPI library will have no way of detecting this, nor should this have any affects on the machine. This is exactly the same as MPI_THREAD_FUNNELED. What exactly does it mean with "only one thread will execute?" The openmpi library has absolutely zero way of knowng I've spawned other pthreads, and since these pthreads aren't actually doing MPI communication, I fail to see how this would interfere. Technically, if you call MPI_INIT_THREAD with MPI_THREAD_SINGLE, you have made a promise that you will not create any other threads in your application. There was a time where OSes shipped threaded and non-threaded malloc, for example, so knowing that might be important for that last bit of performance. There are also some obscure corner cases of the memory model of some architectures where you might get unexpected results if you made an MPI Receive call in an thread and accessed that buffer later from another thread, which may require memory barriers inside the implementation, so there could be some differences between SINGLE and FUNNELED due to those barriers. In Open MPI, we'll handle those corner cases whether you init for SINGLE or FUNNELED, so there's really no practical difference for Open MPI, but you're then slightly less portable. I'm asking because I'm using an open_mpi build ontop of infiniband, and the maximum thread mode is MPI_THREAD_SINGLE. That doesn't seem right; which version of Open MPI are you using? Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories
