Michael, You can
mpirun --mca pml_base_verbose 10 --mca btl_base_verbose 10 --mca mtl_base_verbose 10 ... It might show that pml/cm and mtl/psm2 are used. In that case, then yes, the OmniPath library is used even for intra node communications. If this library is optimized for intra node, then it will internally uses shared memory instead of the NIC. You can force mpirun --mca pml ob1 ... And btl/vader (shared memory) will be used for intra node communications ... unless MPI tasks are from different jobs (read MPI_Comm_spawn()) Cheers, Gilles Michael Di Domenico <mdidomeni...@gmail.com> wrote: >i have a user that's claiming when two ranks on the same node want to >talk with each other, they're using the NIC to talk rather then just >talking directly. > >i've never had to test such a scenario. is there a way for me to >prove one way or another whether two ranks are talking through say the >kernel (or however it actually works) or using the nic? > >i didn't set any flags when i compiled openmpi to change this. > >i'm running ompi 3.1, pmix 2.2.1, and slurm 18.05 running atop omnipath >_______________________________________________ >users mailing list >users@lists.open-mpi.org >https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
