Hi, I've noticed some performance differences when using mpirun and SLURM for job startup. Below I've included example output from the OSU bidirectional bandwidth benchmark that seem to show a significant difference in bandwidth for larger message sizes. I've looked at the OpenMPI FAQ for running jobs under SLURM and my OpenMPI installation was configured using both --with-slurm and --with-pmi. Similar differences in performance seem to happen if you use the "wrong" mpirun executable, for example using the 1.8.4 mpirun to start an executable compiled using 1.8.5 (also shown below). I thought maybe the environment variables set by the mpirun from 1.8.5 were important, but using the same environment variables set by the 1.8.5 version of mpirun didn't seem to influence the behavior. Is this a known issue, or are there ideas of why using SLURM seems to result in such different performance for larger message sizes?
Thanks, Patrick *SLURM:* plegresl@svail-1:~/SRC/openmpi-1.8.5/majel/bin$ *env | grep OMPI* plegresl@svail-1:~/SRC/openmpi-1.8.5/majel/bin$ *srun -p GTX980x4sys --ntasks 2 ./osu_bibw* cpu_bind=MASK - svail-15, task 0 0 [6916]: mask 0x8000 set cpu_bind=MASK - svail-15, task 1 1 [6917]: mask 0x80000000 set # OSU MPI-CUDA Bi-Directional Bandwidth Test # Send Buffer on HOST (H) and Receive Buffer on HOST (H) # Size Bi-Bandwidth (MB/s) 1 1.44 2 6.12 4 12.19 8 24.45 16 48.95 32 97.76 64 193.11 128 375.00 256 688.26 512 1101.45 1024 1975.76 2048 3298.67 4096 3270.65 8192 4737.39 16384 5479.92 32768 6331.54 65536 6392.04 131072 5034.20 262144 3762.04 524288 4543.68 1048576 4244.62 2097152 4233.64 4194304 4232.11 plegresl@svail-1:~/SRC/openmpi-1.8.5/majel/bin$ *export OMPI_MCA_btl=self,vader* plegresl@svail-1:~/SRC/openmpi-1.8.5/majel/bin$ *srun -p GTX980x4sys --ntasks 2 ./osu_bibw* cpu_bind=MASK - svail-15, task 0 0 [6961]: mask 0x8000 set cpu_bind=MASK - svail-15, task 1 1 [6962]: mask 0x80000000 set # OSU MPI-CUDA Bi-Directional Bandwidth Test # Send Buffer on HOST (H) and Receive Buffer on HOST (H) # Size Bi-Bandwidth (MB/s) 1 2.59 2 4.78 4 9.89 8 41.75 16 81.89 32 166.25 64 239.32 128 388.07 256 667.63 512 1166.52 1024 1978.11 2048 3462.92 4096 3831.92 8192 5722.75 16384 8295.46 32768 10401.27 65536 11181.00 131072 10024.32 262144 9953.43 524288 10237.18 1048576 13484.74 2097152 13548.25 4194304 12401.71 plegresl@svail-1:~/SRC/openmpi-1.8.5/majel/bin$ *mpirun:* plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ *env | grep OMPI* plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ */tools/openmpi-1.8.5/bin/mpirun -np 2 --report-bindings -npersocket 1 ./osu_bibw* [svail-15:06868] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..] [svail-15:06868] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..] # OSU MPI-CUDA Bi-Directional Bandwidth Test # Send Buffer on HOST (H) and Receive Buffer on HOST (H) # Size Bi-Bandwidth (MB/s) 1 2.40 2 5.11 4 11.96 8 24.68 16 49.17 32 92.79 64 179.02 128 317.15 256 484.08 512 920.96 1024 1543.13 2048 2588.79 4096 3267.20 8192 4324.19 16384 5642.14 32768 6422.15 65536 6988.72 131072 6793.10 262144 6284.13 524288 6398.42 1048576 7267.68 2097152 7430.67 4194304 7215.42 plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ *export OMPI_MCA_btl=self,vader* plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ */tools/openmpi-1.8.5/bin/mpirun -np 2 --report-bindings -npersocket 1 ./osu_bibw* [svail-15:06892] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..] [svail-15:06892] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..] # OSU MPI-CUDA Bi-Directional Bandwidth Test # Send Buffer on HOST (H) and Receive Buffer on HOST (H) # Size Bi-Bandwidth (MB/s) 1 3.09 2 6.32 4 29.89 8 60.45 16 86.52 32 206.86 64 265.10 128 348.81 256 507.25 512 856.77 1024 1468.91 2048 2508.12 4096 3949.40 8192 6830.64 16384 10118.59 32768 14690.76 65536 18497.65 131072 20405.31 262144 19300.26 524288 18829.65 1048576 19374.49 2097152 21596.29 4194304 24355.79 plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ *"wrong" mpirun:* plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ *env | grep OMPI* OMPI_MCA_btl=self,vader plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$ */tools/openmpi-1.8.4/bin/mpirun -np 2 --report-bindings -npersocket 1 ./osu_bibw* [svail-15:07109] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: [../../../../../../../..][BB/../../../../../../..] [svail-15:07109] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: [BB/../../../../../../..][../../../../../../../..] # OSU MPI-CUDA Bi-Directional Bandwidth Test # Send Buffer on HOST (H) and Receive Buffer on HOST (H) # Size Bi-Bandwidth (MB/s) 1 6.10 2 13.67 4 26.74 8 57.17 16 81.82 32 238.84 64 241.87 128 319.86 256 490.38 512 805.50 1024 1394.54 2048 2350.89 4096 2845.86 8192 3692.30 16384 5196.64 32768 6018.54 65536 6360.56 131072 6540.06 262144 6860.57 524288 7945.36 1048576 7983.90 2097152 7926.69 4194304 7836.43 plegresl@svail-15:~/SRC/openmpi-1.8.5/majel/bin$
