Received from Rolf vandeVaart on Fri, Mar 27, 2015 at 04:09:58PM EDT: > >-----Original Message----- > >From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Lev Givon > >Sent: Friday, March 27, 2015 3:47 PM > >To: us...@open-mpi.org > >Subject: [OMPI users] segfault during MPI_Isend when transmitting GPU > >arrays between multiple GPUs > > > >I'm using PyCUDA 2014.1 and mpi4py (git commit 3746586, uploaded today) > >built against OpenMPI 1.8.4 with CUDA support activated to asynchronously > >send GPU arrays between multiple Tesla GPUs (Fermi generation). Each MPI > >process is associated with a single GPU; the process has a run loop that > >starts > >several Isends to transmit the contents of GPU arrays to destination > >processes and several Irecvs to receive data from source processes into GPU > >arrays on the process' GPU. Some of the sends/recvs use one tag, while the > >remainder use a second tag. A single Waitall invocation is used to wait for > >all of > >these sends and receives to complete before the next iteration of the loop > >can commence. All GPU arrays are preallocated before the run loop starts. > >While this pattern works most of the time, it sometimes fails with a segfault > >that appears to occur during an Isend:
(snip) > >Any ideas as to what could be causing this problem? > > > >I'm using CUDA 6.5-14 with NVIDIA drivers 340.29 on Ubuntu 14.04. > > Hi Lev: > > I am not sure what is happening here but there are a few things we can do to > try and narrow things done. > 1. If you run with --mca btl_smcuda_use_cuda_ipc 0 then I assume this error > will go away? Yes - that appears to be the case. > 2. Do you know if when you see this error it happens on the first pass through > your communications? That is, you mention how there are multiple > iterations through the loop and I am wondering when you see failures if it > is the first pass through the loop. When the segfault occurs, it appears to always happen during the second iteration of the loop, i.e., at least one slew of Isends (and presumably Irecvs) is successfully performed. Some more details regarding the Isends: each process starts two Isends for each destination process to which it transmits data. The Isends use two different tags, respectively; one is passed None (by design), while the other is passed the pointer to a GPU array with nonzero length. The segfault appears to occur during the latter Isend. -- Lev Givon Bionet Group | Neurokernel Project http://www.columbia.edu/~lev/ http://lebedov.github.io/ http://neurokernel.github.io/
