On May 20, 2011, at 03:25 , David Büttner wrote:
> Hello,
>
> thanks for the quick answer. I am sorry that I forgot to mention this: I did
> compile OpenMPI with MPI_THREAD_MULTIPLE support and test if required ==
> provided after the MPI_Thread_init call.
>
>> I do not see any mechanism for protecting the accesses to the requests to a
>> single thread? What is the thread model you're using?
>>
> Again I am sorry that this was not clear: In the pseudo code below I wanted
> to indicate the access-protection I do by thread-id dependent calls if(0 ==
> thread-id) and by using the trylock(...) (using pthread-mutexes). In the code
> all accesses concerning one MPI_Request (which are pthread-global-pointers in
> my case) are protected and called in sequential order, i.e. MPI_Isend/recv is
> returns before any thread is allowed to call the corresponding MPI_Test and
> no-one can call MPI_Test any more when a thread is allowed to call MPI_Wait.
If all these are true the code is then supposed to work. We have multi-threaded
software, that uses a non multi-threaded version of MPI (Open MPI in this
instance), for overlapping communications and computations. Basically what
we're doing is very similar to what you described above, except we ensure no
two threads are accessing __any__ MPI functions in same time. And the code
works perfectly.
> I did this in the same manner before with other MPI implementations, but also
> on the same machine with the same (untouched) OpenMPI implementation, also
> using pthreads and MPI in combination, but I used
>
> MPI_Request req;
>
> instead of
>
> MPI_Request* req;
> (and later)
> req = (MPI_Request*)malloc(sizeof(MPI_Request));
>
>
> In my recent (problem) code, I also tried not using pointers, but got the
> same problem. Also, as I described in the first mail, I tried everything
> concerning the memory allocation of the MPI_Request objects.
> I tried not calling malloc. This I guessed wouldn't work, but the OpenMPI
> documentation says this:
>
> " Nonblocking calls allocate a communication request object and associate it
> with the request handle the argument request). "
> [http://www.open-mpi.org/doc/v1.4/man3/MPI_Isend.3.php] and
>
> " [...] if the communication object was created by a nonblocking send or
> receive, then it is deallocated and the request handle is set to
> MPI_REQUEST_NULL." [http://www.open-mpi.org/doc/v1.4/man3/MPI_Test.3.php] and
> (in slightly different words)
> [http://www.open-mpi.org/doc/v1.4/man3/MPI_Wait.3.php]
>
> So I thought that it might do some kind of optimized memory stuff internally.
>
> I also tried allocating req (for each used MPI_Request) once before the first
> use and deallocation after the last use (which I thought was the way it was
> supposed to work), but that crashes also.
>
> I tried replacing the pointers through global variables
>
> MPI_Request req;
>
> which didn't do the job...
>
> The only thing that seems to work is what I mentioned below: Allocate every
> time I am going to need it in the MPI_Isend/recv, use it in MPI_Test/Wait and
> after that deallocate it by hand each time.
> I don't think that this is supposed to be like this since I have to do a call
> to malloc and free so often (for multiple MPI_Request objects in each
> iteration) that it will most likely limit performance...
I would really recheck the code that make sure that multiple threads cannot
complete a request in same time (MPI_Wait and MPI_Test on the same request on
two threads). Second, I will declare the MPI_Request as volatile, to forbid the
compiler to optimize the accesses to it.
> Anyway I still have the same problem and am still unclear on what kind of
> memory allocation I should be doing for the MPI_Requests. Is there anything
> else (besides MPI_THREAD_MULTIPLE support, thread access control, sequential
> order of MPI_Isend/recv, MPI_Test and MPI_Wait for one MPI_Request object) I
> need to take care of? If not, what could I do to find the source of my
> problem?
If what I proposed above doesn't work, I will go for a thread correctness
checker: valgrind, the intel thread checker or Thread Sanitizer
(http://code.google.com/p/data-race-test/wiki/ThreadSanitizer).
george.
>
> Thanks again for any kind of help!
>
> Kind regards,
> David
>
>
>
>> > From an implementation perspective, your code is correct only if you
>> > initialize the MPI library with MPI_THREAD_MULTIPLE and if the library
>> > accepts. Otherwise, there is an assumption that the application is single
>> > threaded, or that the MPI behavior is implementation dependent. Please
>> > read the MPI standard regarding to MPI_Init_thread for more details.
>>
>> Regards,
>> george.
>>
>> On May 19, 2011, at 02:34 , David Büttner wrote:
>>
>>> Hello,
>>>
>>> I am working on a hybrid MPI (OpenMPI 1.4.3) and Pthread code. I am using
>>> MPI_Isend and MPI_Irecv for communication and MPI_Test/MPI_Wait to check if
>
