Can you try to run the example code "ring_c" across nodes? Josh
On Thu, Aug 14, 2014 at 3:14 PM, Maxime Boissonneault < maxime.boissonnea...@calculquebec.ca> wrote: > Yes, > Everything has been built with GCC 4.8.x, although x might have changed > between the OpenMPI 1.8.1 build and the gromacs build. For OpenMPI 1.8.2rc4 > however, it was the exact same compiler for everything. > > Maxime > > Le 2014-08-14 14:57, Joshua Ladd a écrit : > > Hmmm...weird. Seems like maybe a mismatch between libraries. Did you > build OMPI with the same compiler as you did GROMACS/Charm++? > > I'm stealing this suggestion from an old Gromacs forum with essentially > the same symptom: > > "Did you compile Open MPI and Gromacs with the same compiler (i.e. both > gcc and the same version)? You write you tried different OpenMPI versions > and different GCC versions but it is unclear whether those match. Can you > provide more detail how you compiled (including all options you specified)? > Have you tested any other MPI program linked against those Open MPI > versions? Please make sure (e.g. with ldd) that the MPI and pthread library > you compiled against is also used for execution. If you compiled and run on > different hosts, check whether the error still occurs when executing on the > build host." > > http://redmine.gromacs.org/issues/1025 > > Josh > > > > > On Thu, Aug 14, 2014 at 2:40 PM, Maxime Boissonneault < > maxime.boissonnea...@calculquebec.ca> wrote: > >> I just tried Gromacs with two nodes. It crashes, but with a different >> error. I get >> [gpu-k20-13:142156] *** Process received signal *** >> [gpu-k20-13:142156] Signal: Segmentation fault (11) >> [gpu-k20-13:142156] Signal code: Address not mapped (1) >> [gpu-k20-13:142156] Failing at address: 0x8 >> [gpu-k20-13:142156] [ 0] /lib64/libpthread.so.0(+0xf710)[0x2ac5d070c710] >> [gpu-k20-13:142156] [ 1] >> /usr/lib64/nvidia/libcuda.so.1(+0x263acf)[0x2ac5ddfbcacf] >> [gpu-k20-13:142156] [ 2] >> /usr/lib64/nvidia/libcuda.so.1(+0x229a83)[0x2ac5ddf82a83] >> [gpu-k20-13:142156] [ 3] >> /usr/lib64/nvidia/libcuda.so.1(+0x15b2da)[0x2ac5ddeb42da] >> [gpu-k20-13:142156] [ 4] >> /usr/lib64/nvidia/libcuda.so.1(cuInit+0x43)[0x2ac5ddea0933] >> [gpu-k20-13:142156] [ 5] >> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(+0x15965)[0x2ac5d0930965] >> [gpu-k20-13:142156] [ 6] >> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(+0x15a0a)[0x2ac5d0930a0a] >> [gpu-k20-13:142156] [ 7] >> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(+0x15a3b)[0x2ac5d0930a3b] >> [gpu-k20-13:142156] [ 8] >> /software-gpu/cuda/6.0.37/lib64/libcudart.so.6.0(cudaDriverGetVersion+0x4a)[0x2ac5d094602a] >> [gpu-k20-13:142156] [ 9] >> /software-gpu/apps/gromacs/4.6.5_gcc/lib/libgmxmpi.so.8(gmx_print_version_info_gpu+0x55)[0x2ac5cf9a90b5] >> [gpu-k20-13:142156] [10] >> /software-gpu/apps/gromacs/4.6.5_gcc/lib/libgmxmpi.so.8(gmx_log_open+0x17e)[0x2ac5cf54b9be] >> [gpu-k20-13:142156] [11] mdrunmpi(cmain+0x1cdb)[0x43b4bb] >> [gpu-k20-13:142156] [12] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac5d1534d1d] >> [gpu-k20-13:142156] [13] mdrunmpi[0x407be1] >> [gpu-k20-13:142156] *** End of error message *** >> -------------------------------------------------------------------------- >> mpiexec noticed that process rank 0 with PID 142156 on node gpu-k20-13 >> exited on signal 11 (Segmentation fault). >> -------------------------------------------------------------------------- >> >> >> >> We do not have MPI_THREAD_MULTIPLE enabled in our build, so Charm++ >> cannot be using this level of threading. The configure line for OpenMPI was >> ./configure --prefix=$PREFIX \ >> --with-threads --with-verbs=yes --enable-shared --enable-static \ >> --with-io-romio-flags="--with-file-system=nfs+lustre" \ >> --without-loadleveler --without-slurm --with-tm \ >> --with-cuda=$(dirname $(dirname $(which nvcc))) >> >> Maxime >> >> >> Le 2014-08-14 14:20, Joshua Ladd a écrit : >> >> What about between nodes? Since this is coming from the OpenIB BTL, >> would be good to check this. >> >> Do you know what the MPI thread level is set to when used with the >> Charm++ runtime? Is it MPI_THREAD_MULTIPLE? The OpenIB BTL is not thread >> safe. >> >> Josh >> >> >> On Thu, Aug 14, 2014 at 2:17 PM, Maxime Boissonneault < >> maxime.boissonnea...@calculquebec.ca> wrote: >> >>> Hi, >>> I ran gromacs successfully with OpenMPI 1.8.1 and Cuda 6.0.37 on a >>> single node, with 8 ranks and multiple OpenMP threads. >>> >>> Maxime >>> >>> >>> Le 2014-08-14 14:15, Joshua Ladd a écrit : >>> >>> Hi, Maxime >>> >>> Just curious, are you able to run a vanilla MPI program? Can you try >>> one one of the example programs in the "examples" subdirectory. Looks like >>> a threading issue to me. >>> >>> Thanks, >>> >>> Josh >>> >>> >>> >>> _______________________________________________ >>> users mailing listus...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/08/25023.php >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/08/25024.php >>> >> >> >> >> _______________________________________________ >> users mailing listus...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/08/25025.php >> >> >> >> -- >> --------------------------------- >> Maxime Boissonneault >> Analyste de calcul - Calcul Québec, Université Laval >> Ph. D. en physique >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/08/25026.php >> > > > > _______________________________________________ > users mailing listus...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/08/25027.php > > > > -- > --------------------------------- > Maxime Boissonneault > Analyste de calcul - Calcul Québec, Université Laval > Ph. D. en physique > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/08/25028.php >
