Victor, If the MX library supports a mapper, Open MPI takes provide selective hardware activation. Look at the MCA parameters supported by the MX devices to get more info (ompi_info —mca btl mx). The one that is of interest in this particular case is mx_if_include, allowing MX based jobs to only use the Myrinet card that has a mapper matching the provided key. You should set this MCA parameter to the last 6 digits of your mapper MAC (—mca btl mx_if_include abcdef).
George. On Jan 14, 2014, at 00:36 , Barrett, Brian W <bwba...@sandia.gov> wrote: > Victor - > > I don't think our multi-port support with MX is particularly well tested (I > know I don't test that path). > > It looks like you might be able to work around the problem by setting -mca > mtl_mx_endpoint_num 1 on the mpirun command line, which will only use the > first port found. But I could be wrong. > > Brian > > On 1/9/14 5:02 PM, "Victor Prosolin" <victor.proso...@rwdi.com> wrote: > >> H, >> Our cluster has servers with either a single port or a dual port Myrinet >> card. In case of a dual card, only one port is connected to the Myrinet >> switch. The OpenMPI library is configured with “--with-mx=…” option and it >> works fine when I submit jobs to single port servers only. However, when I >> try to include a server with a dual port card, I get a bunch of errors like >> the following: >> [compute-08:17788] mx_connect fail for unknown 60dd464f9d nic_id with key >> aaaaffff (error Destination NIC not found in network table) >> >> 60dd464f9d is the wrong MAC address corresponding to port 1 (not connected) >> when port 0 is connected to the switch and has MAC 60dd464f9c. >> >> This is how (try to) I run the job: >> 1. mpiexec -np 32 -host compute-08,compute-17,compute-18,compute-16 >> -mca mtl mx --mca pml cm ./wrf.exe >> or >> 2. Using a similar command but via Sun Grid Engine. >> >> The OS is Centos 6.4, 64bit. OpenMPI 1.6.5 compiled from the official src >> rpm with gcc 4.4.7, MX library 1.2.16 manually compiled. Again, this >> configuration works fine when only single port servers are used. >> >> Is there a way to tell OpenMPI to stick to the one port that is connected? I >> haven’t found any options through ompi_info or via google… Any help will be >> greatly appreciated. >> >> Sincerely, >> Victor. >> > > > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
