And maybe include your LD_LIBRARY_PATH Josh
On Thu, Aug 14, 2014 at 3:16 PM, Joshua Ladd <jladd.m...@gmail.com> wrote: > Can you try to run the example code "ring_c" across nodes? > > Josh > > > On Thu, Aug 14, 2014 at 3:14 PM, Maxime Boissonneault < > maxime.boissonnea...@calculquebec.ca> wrote: > >> Yes, >> Everything has been built with GCC 4.8.x, although x might have changed >> between the OpenMPI 1.8.1 build and the gromacs build. For OpenMPI 1.8.2rc4 >> however, it was the exact same compiler for everything. >> >> Maxime >> >> Le 2014-08-14 14:57, Joshua Ladd a écrit : >> >> Hmmm...weird. Seems like maybe a mismatch between libraries. Did you >> build OMPI with the same compiler as you did GROMACS/Charm++? >> >> I'm stealing this suggestion from an old Gromacs forum with essentially >> the same symptom: >> >> "Did you compile Open MPI and Gromacs with the same compiler (i.e. both >> gcc and the same version)? You write you tried different OpenMPI versions >> and different GCC versions but it is unclear whether those match. Can you >> provide more detail how you compiled (including all options you specified)? >> Have you tested any other MPI program linked against those Open MPI >> versions? Please make sure (e.g. with ldd) that the MPI and pthread library >> you compiled against is also used for execution. If you compiled and run on >> different hosts, check whether the error still occurs when executing on the >> build host." >> >> http://redmine.gromacs.org/issues/1025 >> >> Josh >> >> >> >> >> On Thu, Aug 14, 2014 at 2:40 PM, Maxime Boissonneault < >> maxime.boissonnea...@calculquebec.ca> wrote: >> >>> I just tried Gromacs with two nodes. It crashes, but with a different >>> error. I get >>> [gpu-k20-13:142156] *** Process received signal *** >>> [gpu-k20-13:142156] Signal: Segmentation fault (11) >>> [gpu-k20-13:142156] Signal code: Address not mapped (1) >>> [gpu-k20-13:142156] Failing at address: 0x8 >>> [gpu-k20-13:142156] [ 0] /lib64/libpthread.so.0(+0xf710)[0x2ac5d070c710] >>> [gpu-k20-13:142156] [ 1] >>> /usr/lib64/nvidia/libcuda.so.1(+0x263acf)[0x2ac5ddfbcacf] >>> [gpu-k20-13:142156] [ 2] >>> /usr/lib64/nvidia/libcuda.so.1(+0x229a83)[0x2ac5ddf82a83] >>> [gpu-k20-13:142156] [ 3] >>> /usr/lib64/nvidia/libcuda.so.1(+0x15b2da)[0x2ac5ddeb42da] >>> [gpu-k20-13:142156] [ 4] >>> /usr/lib64/nvidia/libcuda.so.1(cuInit+0x43)[0x2ac5ddea0933] >>> [gpu-k20-13:142156] [ 5] >>> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(+0x15965)[0x2ac5d0930965] >>> [gpu-k20-13:142156] [ 6] >>> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(+0x15a0a)[0x2ac5d0930a0a] >>> [gpu-k20-13:142156] [ 7] >>> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(+0x15a3b)[0x2ac5d0930a3b] >>> [gpu-k20-13:142156] [ 8] >>> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(cudaDriverGetVersion+0x4a)[0x2ac5d094602a] >>> [gpu-k20-13:142156] [ 9] >>> /software-gpu/apps/gromacs/4.6.5_gcc/lib/libgmxmpi.so.8(gmx_print_version_info_gpu+0x55)[0x2ac5cf9a90b5] >>> [gpu-k20-13:142156] [10] >>> /software-gpu/apps/gromacs/4.6.5_gcc/lib/libgmxmpi.so.8(gmx_log_open+0x17e)[0x2ac5cf54b9be] >>> [gpu-k20-13:142156] [11] mdrunmpi(cmain+0x1cdb)[0x43b4bb] >>> [gpu-k20-13:142156] [12] >>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac5d1534d1d] >>> [gpu-k20-13:142156] [13] mdrunmpi[0x407be1] >>> [gpu-k20-13:142156] *** End of error message *** >>> >>> -------------------------------------------------------------------------- >>> mpiexec noticed that process rank 0 with PID 142156 on node gpu-k20-13 >>> exited on signal 11 (Segmentation fault). >>> >>> -------------------------------------------------------------------------- >>> >>> >>> >>> We do not have MPI_THREAD_MULTIPLE enabled in our build, so Charm++ >>> cannot be using this level of threading. The configure line for OpenMPI was >>> ./configure --prefix=$PREFIX \ >>> --with-threads --with-verbs=yes --enable-shared --enable-static \ >>> --with-io-romio-flags="--with-file-system=nfs+lustre" \ >>> --without-loadleveler --without-slurm --with-tm \ >>> --with-cuda=$(dirname $(dirname $(which nvcc))) >>> >>> Maxime >>> >>> >>> Le 2014-08-14 14:20, Joshua Ladd a écrit : >>> >>> What about between nodes? Since this is coming from the OpenIB BTL, >>> would be good to check this. >>> >>> Do you know what the MPI thread level is set to when used with the >>> Charm++ runtime? Is it MPI_THREAD_MULTIPLE? The OpenIB BTL is not thread >>> safe. >>> >>> Josh >>> >>> >>> On Thu, Aug 14, 2014 at 2:17 PM, Maxime Boissonneault < >>> maxime.boissonnea...@calculquebec.ca> wrote: >>> >>>> Hi, >>>> I ran gromacs successfully with OpenMPI 1.8.1 and Cuda 6.0.37 on a >>>> single node, with 8 ranks and multiple OpenMP threads. >>>> >>>> Maxime >>>> >>>> >>>> Le 2014-08-14 14:15, Joshua Ladd a écrit : >>>> >>>> Hi, Maxime >>>> >>>> Just curious, are you able to run a vanilla MPI program? Can you try >>>> one one of the example programs in the "examples" subdirectory. Looks like >>>> a threading issue to me. >>>> >>>> Thanks, >>>> >>>> Josh >>>> >>>> >>>> >>>> _______________________________________________ >>>> users mailing listus...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/08/25023.php >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/08/25024.php >>>> >>> >>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/08/25025.php >>> >>> >>> >>> -- >>> --------------------------------- >>> Maxime Boissonneault >>> Analyste de calcul - Calcul Québec, Université Laval >>> Ph. D. en physique >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/08/25026.php >>> >> >> >> >> _______________________________________________ >> users mailing listus...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/08/25027.php >> >> >> >> -- >> --------------------------------- >> Maxime Boissonneault >> Analyste de calcul - Calcul Québec, Université Laval >> Ph. D. en physique >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/08/25028.php >> > >
