It is dangerous to hold a local lock (like a mutex} across a blocking MPI
call unless you can be 100% sure everything that must happen remotely will
be completely independent of what is done with local locks & communication
dependancies on other tasks.
It is likely that a MPI_Comm_spawn call in which the spawning communicator
is MPI_COMM_SELF would be safe to serialize with a mutex. But be careful
and do not view this as an approach to making MPI applications thread safe
in general. Also, unless you are calling MPI_Comm_spawn on a single task
communicator you would need to have a different input communicator for each
thread that will make an MPI_Comm_spawn call. MPI requires that collective
calls on a given communicator be made in the same order by all
participating tasks.
If there are two or more tasks making the MPI_Comm_spawn call collectively
from multiple threads (even with per-thread input communicators) then using
a local lock this way is pretty sure to deadlock at some point. Say task 0
serializes spawning threads as A then B and task 1 serializes them as B
then A. The job will deadlock because task 0 cannot free its lock for
thread A until task 1 makes the spawn call for thread A as well. That will
never happen if task 1 is stuck in a lock that will not release until task
0 makes its call for thread B.
When you look at the code for a particular task and consider thread
interactions within the task, the use of the lock looks safe. It is only
when you consider the dependancies on what other tasks are doing that the
danger becomes clear. This particular case is pretty easy to see but
sometime when there is a temptation to hold a local mutex across an
blocking MPI call, the chain of dependancies that can lead to deadlock
becomes very hard to predict.
BTW - maybe this is obvious but you also need to protect the logic which
calls MPI_Thread_init to make sure you do not have a a race in which 2
threads each race to test the flag for whether MPI_Init_thread has already
been called. If two thread do:
1) if (MPI_Inited_flag == FALSE) {
2) set MPI_Inited_flag
3) MPI_Init_thread
4) }
You have a couple race conditions.
1) Two threads may both try to call MPI_Iint_thread if one thread tests "
if (MPI_Inited_flag == FALSE)" while the other is between statements 1 & 2.
2) If some thread tests "if (MPI_Inited_flag == FALSE)" while another
thread is between statements 2 and 3, that thread could assume
MPI_Init_thread is done and make the MPI_Comm_spawn call before the thread
that is trying to initialize MPI manages to do it.
Dick
Dick Treumann - MPI Team
IBM Systems & Technology Group
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
Tele (845) 433-7846 Fax (845) 433-8363
users-boun...@open-mpi.org wrote on 09/17/2009 11:36:48 PM:
> [image removed]
>
> Re: [OMPI users] Multi-threading with OpenMPI ?
>
> Ralph Castain
>
> to:
>
> Open MPI Users
>
> 09/17/2009 11:37 PM
>
> Sent by:
>
> users-boun...@open-mpi.org
>
> Please respond to Open MPI Users
>
> Only thing I can suggest is to place a thread lock around the call to
> comm_spawn so that only one thread at a time can execute that
> function. The call to mpi_init_thread is fine - you just need to
> explicitly protect the call to comm_spawn.
>
>
