Hi Iman, How are you handling GPU affinity? Are you using CUDA_VISIBLE_DEVICES for that? If yes can you try using cudaSetDevice in your application instead? Also when multiple processes are assigned to a single GPU are you using MPS and what GPUs are your running this on?
Hope this Helps Jiri > Message: 2 > Date: Wed, 4 May 2016 15:55:20 -0400 > From: Iman Faraji <i.far...@queensu.ca> > To: us...@open-mpi.org > Subject: [OMPI users] Multiple Non-blocking Send/Recv calls with > MPI_Waitall fails when CUDA IPC is in use > Message-ID: > <CADOv2j3_5Exaavj0LUoLeavbZcCYXLTsbnZdLwqGYty2TJd_bQ@mai > l.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi there, > > I am using multiple MPI non-blocking send receives on the GPU buffer > followed by a waitall at the end; I also repeat this process multiple times. > > The MPI version that I am using 1.10.2. > > When multiple processes are assigned to a single GPU (or when CUDA IPC is > used), I get the following error at the beginning > > The call to cuIpcGetEventHandle failed. This is a unrecoverable error and will > cause the program to abort. > cuIpcGetEventHandle return value: 1 > > and this at the end of my benchmark > > The call to cuEventDestory failed. This is a unrecoverable error and will > cause > the program to abort. > cuEventDestory return value: 400 > Check the cuda.h file for what the return value means. > > > *Note1: * > > This error doesn't appear if only one iteration of the non-blocking > send/receive call is used (i.e., using MPI_Waitall only once ) > > This error doesn't appear if multiple iterations are used by MPI_Waitall is > not > included. > > *Note 2:* > > This error doesn't exist if the buffer is is allocated on the host. > > *Note 3:* > > This error doesn't exist if cuda_ipc is disabled or OMPI version 1.8.8 is > used. > > > I'd appreciate if you let me know what causes this issue and how it can be > resolved. > > Regards, > Iman NVIDIA GmbH, Wuerselen, Germany, Amtsgericht Aachen, HRB 8361 Managing Director: Karen Theresa Burns ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
