Can you try upgrading to Open MPI v2.0.2? We just released that last week with a bunch of bug fixes.
> On Feb 7, 2017, at 3:07 PM, Jingchao Zhang <zh...@unl.edu> wrote: > > Hi Tobias, > > Thanks for the reply. I tried both "export OMPI_MCA_mpi_leave_pinned=0" and > "mpirun -mca mpi_leave_pinned 0" but still got the same behavior. Our OpenMPI > version is 2.0.1. Repo version is v2.0.0-257-gee86e07. We have Intel Qlogic > and OPA networks on the same cluster. > > Below is our configuration flags: > ./configure --prefix=$PREFIX \ > --with-hwloc=internal \ > --enable-mpirun-prefix-by-default \ > --with-slurm \ > --with-verbs \ > --with-psm \ > --with-psm2 \ > --disable-openib-connectx-xrc \ > --with-knem=/opt/knem-1.1.2.90mlnx1 \ > --with-cma > > So the question remains why OpenMPI choose openib over self,sm for single > node jobs? Isn't there a mechanism to differentiate btl networks for > single/multi-node jobs? > > Thanks, > Jingchao > From: users <users-boun...@lists.open-mpi.org> on behalf of Tobias Kloeffel > <tobias.kloef...@fau.de> > Sent: Tuesday, February 7, 2017 2:54:46 AM > To: Open MPI Users > Subject: Re: [OMPI users] openmpi single node jobs using btl openib > > Hello Jingchao, > try to use -mca mpi_leave_pinned 0, also for multinode jobs. > > kind regards, > Tobias Klöffel > > On 02/06/2017 09:38 PM, Jingchao Zhang wrote: >> Hi, >> >> We recently noticed openmpi is using btl openib over self,sm for single node >> jobs, which has caused performance degradation for some applications, e.g. >> 'cp2k'. For opempi version 2.0.1, our test shows single node 'cp2k' job >> using openib is ~25% slower than using self,sm. We advise users do '--mca >> btl_base_exclude openib' as a temporary fix. I need to point out that not >> all applications are affected by this feature. Many of them have the same >> single-node performance with/without openib. Why doesn't openmpi use self,sm >> by default for single node jobs? Is this the intended behavior? >> >> Thanks, >> Jingchao >> >> >> _______________________________________________ >> users mailing list >> >> users@lists.open-mpi.org >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > -- > M.Sc. Tobias Klöffel > ======================================================= > Interdisciplinary Center for Molecular Materials (ICMM) > and Computer-Chemistry-Center (CCC) > Department Chemie und Pharmazie > Friedrich-Alexander-Universität Erlangen-Nürnberg > Nägelsbachstr. 25 > D-91052 Erlangen, Germany > > Room: 2.305 > Phone: +49 (0) 9131 / 85 - 20423 > Fax: +49 (0) 9131 / 85 - 26565 > > ======================================================= > > E-mail: > tobias.kloef...@fau.de > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users -- Jeff Squyres jsquy...@cisco.com _______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
