[OMPI users] difference between single and double precision
Dear OpenMPI users I am dealing with an arithmetic problem. In fact, I have two variants of my code: one in single precision, one in double precision. When I compare the two executable built with MPICH, one can observed an expected difference of performance: 115.7-sec in single precision against 178.68-sec in double precision (+54%). The thing is, when I use OpenMPI, the difference is really bigger: 238.5-sec in single precision against 403.19-sec double precision (+69%). Our experiences have already shown OpenMPI is less efficient than MPICH on Ethernet with a small number of processes. This explain the differences between the first set of results with MPICH and the second set with OpenMPI. (But if someone have more information about that or even a solution, I am of course interested.) But, using OpenMPI increases the difference between the two arithmetic. Is it the accentuation of the OpenMPI+Ethernet loss of performance, is it another issue into OpenMPI or is there any option a can use? Thank you for your help. Mathieu. -- Mathieu Gontier
[OMPI users] error mesages appeared but program runs successfully?
Dear list, i am currently try to use the OpenMPI package i install it at my home directory ./configure --prefix=$HOME --enable-mpi-threads make make install and the i add the ~/bin to the path and ~/lib to the ld_library_path to my .bashrc file everything seems normal as i can run the example programs: mpirun -n 8 hello_cxx mpirun -n 8 hello_f77 mpirun -n 8 hello_c etc... but error messages appeas: $ mpirun -n 8 hello_cxx librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. -- [[32727,1],1]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: OpenFabrics (openib) Host: localhost.localdomain Another transport will be used instead, although this may result in lower performance. -- librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. libibverbs: Fatal: couldn't read uverbs ABI version. Hello, world! I am 1 of 8 Hello, world! I am 0 of 8 Hello, world! I am 3 of 8 Hello, world! I am 5 of 8 Hello, world! I am 7 of 8 Hello, world! I am 4 of 8 Hello, world! I am 6 of 8 Hello, world! I am 2 of 8 [localhost.localdomain:30503] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics [localhost.localdomain:30503] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages i am wondering whether i install openmpi the right way anyone would give some suggestions? thanks in advance. Best Regards. -- Daofeng Li College of Biological Science China Agricultural University Beijing China
Re: [OMPI users] Infiniband problem, kernel mismatch
On Friday 19 November 2010 01:03:35 HeeJin Kim wrote: ... > * mlx4: There is a mismatch between the kernel and the userspace > libraries: Kernel does not support XRC. Exiting.* ... > What I'm thinking is that the infiniband card is installed but it doesn't > work in correct mode. > My linux kernel version is *2.6.18-164.el5*, and installed ofed > version is *kernel-ib-pp-1.4.1-ofed20090528r1.4.1sgi605r1.rhel5 Why don't you as a first step try the ib software that is included with EL5.4 (that is, don't install OFED). We run several clusters this way. Also, consider updating to 5.5 (the version you're on includes several security vulnerabilities). /Peter signature.asc Description: This is a digitally signed message part.
Re: [OMPI users] Open MPI vs IBM MPI performance help
Collecting MPI Profile information might help narrow down the issue. You could use some of the tools mentioned here - http://www.open-mpi.org/faq/?category=perftools --Nysal On Wed, Dec 1, 2010 at 11:59 PM, Price, Brian M (N-KCI) < brian.m.pr...@lmco.com> wrote: > OpenMPI version: 1.4.3 > > Platform: IBM P5, 32 processors, 256 GB memory, Symmetric Multi-Threading > (SMT) enabled > > Application: starts up 48 processes and does MPI using MPI_Barrier, > MPI_Get, MPI_Put (lots of transfers, large amounts of data) > > Issue: When implemented using Open MPI vs. IBM’s MPI (‘poe’ from HPC > Toolkit), the application runs 3-5 times slower. > > I suspect that IBM’s MPI implementation must take advantage of some > knowledge that it has about data transfers that Open MPI is not taking > advantage of. > > Any suggestions? > > Thanks, > > Brian Price > > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] error mesages appeared but program runs successfully?
It means that you probably have a version mismatch with your OpenFabrics drivers and or you have no OpenFabrics hardware and you should probably disable those drivers. Sent from my PDA. No type good. On Dec 3, 2010, at 4:56 AM, "Daofeng Li" wrote: > Dear list, > > i am currently try to use the OpenMPI package > i install it at my home directory > ./configure --prefix=$HOME --enable-mpi-threads > make > make install > > and the i add the ~/bin to the path and ~/lib to the ld_library_path to my > .bashrc file > > everything seems normal as i can run the example programs: > mpirun -n 8 hello_cxx > mpirun -n 8 hello_f77 > mpirun -n 8 hello_c > etc... > > but error messages appeas: > > $ mpirun -n 8 hello_cxx > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > -- > [[32727,1],1]: A high-performance Open MPI point-to-point messaging module > was unable to find any relevant network interfaces: > Module: OpenFabrics (openib) > Host: localhost.localdomain > Another transport will be used instead, although this may result in > lower performance. > -- > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > libibverbs: Fatal: couldn't read uverbs ABI version. > Hello, world! I am 1 of 8 > Hello, world! I am 0 of 8 > Hello, world! I am 3 of 8 > Hello, world! I am 5 of 8 > Hello, world! I am 7 of 8 > Hello, world! I am 4 of 8 > Hello, world! I am 6 of 8 > Hello, world! I am 2 of 8 > [localhost.localdomain:30503] 7 more processes have sent help message > help-mpi-btl-base.txt / btl:no-nics > [localhost.localdomain:30503] Set MCA parameter "orte_base_help_aggregate" to > 0 to see all help / error messages > > i am wondering whether i install openmpi the right way > anyone would give some suggestions? > > thanks in advance. > > Best Regards. > -- > Daofeng Li > College of Biological Science > China Agricultural University > Beijing > China > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] difference between single and double precision
Yes, we have never really optimized open MPI for tcp. That is changing soon, hopefully. Regardless, what is the communication pattern of your app? Are you sending a lot of data frequently? Even the MPICH perf difference is surprising - it suggests a lot of data xfer, potentially with small messages...? Another option to try is to install the openmx drivers on your system and run open MPI with mx support. This should be much better perf than tcp. Sent from my PDA. No type good. On Dec 3, 2010, at 3:11 AM, "Mathieu Gontier" wrote: > > Dear OpenMPI users > > I am dealing with an arithmetic problem. In fact, I have two variants of my > code: one in single precision, one in double precision. When I compare the > two executable built with MPICH, one can observed an expected difference of > performance: 115.7-sec in single precision against 178.68-sec in double > precision (+54%). > > The thing is, when I use OpenMPI, the difference is really bigger: 238.5-sec > in single precision against 403.19-sec double precision (+69%). > > Our experiences have already shown OpenMPI is less efficient than MPICH on > Ethernet with a small number of processes. This explain the differences > between the first set of results with MPICH and the second set with OpenMPI. > (But if someone have more information about that or even a solution, I am of > course interested.) > But, using OpenMPI increases the difference between the two arithmetic. Is it > the accentuation of the OpenMPI+Ethernet loss of performance, is it another > issue into OpenMPI or is there any option a can use? > > Thank you for your help. > Mathieu. > > -- > Mathieu Gontier > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] difference between single and double precision
On 2010-12-03, at 8:46AM, Jeff Squyres (jsquyres) wrote: > Another option to try is to install the openmx drivers on your system and run > open MPI with mx support. This should be much better perf than tcp. We've tried this on a big GigE cluster (in fact, Brice Goglin was playing with it on our system) -- it's not really an answer. It didn't work past a small number of nodes, and the performance gains were fairly small. IntelMPIs Direct Ethernet Transport did work on larger nodecounts, but again it was a pretty modest effect (few percent decrease in pingpong latencies, no discernable bandwidth improvements). - Jonathan -- Jonathan Dursi SciNet, Compute/Calcul Canada
Re: [OMPI users] error mesages appeared but program runs successfully?
Dear Jeff, actually i didnot understand thiscan you or anyone tell me what to do? Thx. Best. On Fri, Dec 3, 2010 at 9:41 PM, Jeff Squyres (jsquyres) wrote: > It means that you probably have a version mismatch with your OpenFabrics > drivers and or you have no OpenFabrics hardware and you should probably > disable those drivers. > > Sent from my PDA. No type good. > > On Dec 3, 2010, at 4:56 AM, "Daofeng Li" wrote: > > Dear list, > > i am currently try to use the OpenMPI package > i install it at my home directory > ./configure --prefix=$HOME --enable-mpi-threads > make > make install > > and the i add the ~/bin to the path and ~/lib to the ld_library_path to my > .bashrc file > > everything seems normal as i can run the example programs: > mpirun -n 8 hello_cxx > mpirun -n 8 hello_f77 > mpirun -n 8 hello_c > etc... > > but error messages appeas: > > $ mpirun -n 8 hello_cxx > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > -- > [[32727,1],1]: A high-performance Open MPI point-to-point messaging module > was unable to find any relevant network interfaces: > Module: OpenFabrics (openib) > Host: localhost.localdomain > Another transport will be used instead, although this may result in > lower performance. > -- > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > libibverbs: Fatal: couldn't read uverbs ABI version. > CMA: unable to open /dev/infiniband/rdma_cm > librdmacm: couldn't read ABI version. > librdmacm: assuming: 4 > libibverbs: Fatal: couldn't read uverbs ABI version. > libibverbs: Fatal: couldn't read uverbs ABI version. > Hello, world! I am 1 of 8 > Hello, world! I am 0 of 8 > Hello, world! I am 3 of 8 > Hello, world! I am 5 of 8 > Hello, world! I am 7 of 8 > Hello, world! I am 4 of 8 > Hello, world! I am 6 of 8 > Hello, world! I am 2 of 8 > [localhost.localdomain:30503] 7 more processes have sent help message > help-mpi-btl-base.txt / btl:no-nics > [localhost.localdomain:30503] Set MCA parameter "orte_base_help_aggregate" > to 0 to see all help / error messages > > i am wondering whether i install openmpi the right way > anyone would give some suggestions? > > thanks in advance. > > Best Regards. > -- > Daofeng Li > College of Biological Science > China Agricultural University > Beijing > China > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- Daofeng Li College of Biological Science China Agricultural University Beijing China
Re: [OMPI users] error mesages appeared but program runs successfully?
Hi Daofeng Do you have an Infiniband card in the machine where you are running the program? (Open Fabrics / OFED is the software support for Infiniband. I guess you need the same version installed in all machines.) Does the directory referred in the error message actually exist in your machine (i.e, /dev/infiniband) ? Are you running it in the same machine where you installed OpenMPI? What output do you get from: /usr/sbin/ibstat ? Did you compile the programs with the mpicc,mpiCC, mpif77 from the same OpenMPI that you built? (Some Linux distributions and compilers come with their own flavors of MPI, or you may also have installed MPICH or MVAPICH, so it is not uncommon to mix up.) Have you tried to suppress the use of Infinband, i.e.: mpirun -mca btl ^openib -n 8 hello_cxx (Well, "openib" is the OpenMPI support for Infiniband. The "^" means "don't use it") I hope this helps, Gus Correa Daofeng Li wrote: Dear Jeff, actually i didnot understand thiscan you or anyone tell me what to do? Thx. Best. On Fri, Dec 3, 2010 at 9:41 PM, Jeff Squyres (jsquyres) mailto:jsquy...@cisco.com>> wrote: It means that you probably have a version mismatch with your OpenFabrics drivers and or you have no OpenFabrics hardware and you should probably disable those drivers. Sent from my PDA. No type good. On Dec 3, 2010, at 4:56 AM, "Daofeng Li" mailto:lid...@gmail.com>> wrote: Dear list, i am currently try to use the OpenMPI package i install it at my home directory ./configure --prefix=$HOME --enable-mpi-threads make make install and the i add the ~/bin to the path and ~/lib to the ld_library_path to my .bashrc file everything seems normal as i can run the example programs: mpirun -n 8 hello_cxx mpirun -n 8 hello_f77 mpirun -n 8 hello_c etc... but error messages appeas: $ mpirun -n 8 hello_cxx librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. -- [[32727,1],1]: A high-performance Open MPI point-to-point messaging module was unable to find any relevant network interfaces: Module: OpenFabrics (openib) Host: localhost.localdomain Another transport will be used instead, although this may result in lower performance. -- librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm libibverbs: Fatal: couldn't read uverbs ABI version. CMA: unable to open /dev/infiniband/rdma_cm librdmacm: couldn't read ABI version. librdmacm: assuming: 4 libibverbs: Fatal: couldn't read uverbs ABI version. libibverbs: Fatal: couldn't read uverbs ABI version. Hello, world! I am 1 of 8 Hello, world! I am 0 of 8 Hello, world! I am 3 of 8 Hello, world! I am 5 of 8 Hello, world! I am 7 of 8 Hello, world! I am 4 of 8 Hello, world! I am 6 of 8 Hello, world! I am 2 of 8 [localhost.localdomain:30503] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics [localhost.localdomain:30503] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages i am wondering whether i install openmpi the right way anyone would give some suggestions? thanks in advance. Best Regards. -- Daofeng Li College of Biological Science China Agricultural University Be
