With Open MPI 1.8.1, the library will use the NIC that is "closest" to the CPU. There was a bug in earlier versions of Open MPI 1.8 so that did not happen. You can see this by running with some verbosity using the "btl_base_verbose" flag. For example, this is what I observed on a two node cluster with two NICs on each node.
[rvandevaart@ivy0] $ mpirun --mca btl_base_verbose 1 -host ivy0,ivy1 -np 4 --mca pml ob1 --mca btl_openib_warn_default_gid_prefix 0 MPI_Alltoall_c [ivy0.nvidia.com:28896] [rank=0] openib: using device mlx5_0 [ivy0.nvidia.com:28896] [rank=0] openib: skipping device mlx5_1; it is too far away [ivy0.nvidia.com:28897] [rank=1] openib: using device mlx5_1 [ivy0.nvidia.com:28897] [rank=1] openib: skipping device mlx5_0; it is too far away [ivy1.nvidia.com:04652] [rank=2] openib: using device mlx5_0 [ivy1.nvidia.com:04652] [rank=2] openib: skipping device mlx5_1; it is too far away [ivy1.nvidia.com:04653] [rank=3] openib: using device mlx5_1 [ivy1.nvidia.com:04653] [rank=3] openib: skipping device mlx5_0; it is too far away So, maybe the right thing is happening by default? Or you are looking for more fine-grained control? Rolf ________________________________________ From: users [users-boun...@open-mpi.org] On Behalf Of Tobias Kloeffel [tobias.kloef...@fau.de] Sent: Sunday, July 20, 2014 12:33 PM To: Open MPI Users Subject: Re: [OMPI users] Help with multirail configuration I found no option in 1.6.5 and 1.8.1... Am 7/20/2014 6:29 PM, schrieb Ralph Castain: > What version of OMPI are you talking about? > > On Jul 20, 2014, at 9:11 AM, Tobias Kloeffel <tobias.kloef...@fau.de> wrote: > >> Hello everyone, >> >> I am trying to get the maximum performance out of my two node testing setup. >> Each node consists of 4 Sandy Bridge CPUs and each CPU has one directly >> attached Mellanox QDR card. Both nodes are connected via a 8-port Mellanox >> switch. >> So far I found no option that allows binding mpi-ranks to a specific card, >> as it is available in MVAPICH2. Is there a way to change the round robin >> behavior of openMPI? >> Maybe something like "btl_tcp_if_seq" that I have missed? >> >> >> Kind regards, >> Tobias >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/07/24822.php > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/07/24825.php _______________________________________________ users mailing list us...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2014/07/24827.php ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
