A little more info for you:
When I constrain 'srun' with 'ntasks-per-node', it seems to do OK . . .? I'm forwarding this to slurm-dev. SD: I'm having a problem using srun -N 2 -n 8, as you can see below . . . -------- Forwarded Message -------- Subject: segfault on certain configuration Date: Wed, 24 May 2017 16:28:21 -0700 From: Geof Sawaya <[email protected]><mailto:[email protected]> To: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]> Hi MVAPICH people, I have another problem. I am using the GDR (2.2) version with Slurm 17. I have this simple MPI program that segfaults occasionally (depending upon the number of processes per node). Here is a small case, where I use srun -N 2 -n 8 (2 nodes, 8 tasks) and get: 2 procs on node 2, and 6 on node 1. Rank 5 crashes. Here's the trace: Program received signal SIGSEGV, Segmentation fault. 0x00007f5c28128a12 in post_srq_send () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 (gdb) bt #0 0x00007f5c28128a12 in post_srq_send () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #1 0x00007f5c2812d532 in vbuf_address_send () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #2 0x00007f5c2812e8f9 in MPIDI_CH3I_MRAIL_Parse_header () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #3 0x00007f5c28114a2a in handle_read () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #4 0x00007f5c2811519b in MPIDI_CH3I_Progress () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #5 0x00007f5c280be09d in MPIC_Wait () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #6 0x00007f5c280be976 in MPIC_Sendrecv () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #7 0x00007f5c27e8c95c in MPIR_Pairwise_Barrier_MV2 () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #8 0x00007f5c27e8caa1 in MPIR_Barrier_intra_MV2 () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #9 0x00007f5c27e8cc89 in MPIR_Barrier_MV2 () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #10 0x00007f5c27e2fc0f in MPIR_Barrier_impl () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #11 0x00007f5c28087f05 in PMPI_Finalize () from /opt/mvapich2/gdr/2.2/cuda8.0/gnu/lib64/libmpi.so.12 #12 0x0000000000401065 in main (argv=1, argc=0x7ffeb2d7dd78) at test.cpp:49 (gdb) Does this give you enough to go on to (hopefully) tell me / guess how I've misconfigured? Many thanks -- Geof
