Re: [OMPI users] Performance: MPICH2 vs OpenMPI
On Fri, Oct 10, 2008 at 10:40 PM, Brian Dobbins wrote: > > Hi guys, > > On Fri, Oct 10, 2008 at 12:57 PM, Brock Palen wrote: > >> Actually I had a much differnt results, >> >> gromacs-3.3.1 one node dual core dual socket opt2218 openmpi-1.2.7 >> pgi/7.2 >> mpich2 gcc >> > >For some reason, the difference in minutes didn't come through, it > seems, but I would guess that if it's a medium-large difference, then it has > its roots in PGI7.2 vs. GCC rather than MPICH2 vs. OpenMPI. Though, to be > fair, I find GCC vs. PGI (for C code) is often a toss-up - one may beat the > other handily on one code, and then lose just as badly on another. > > I think my install of mpich2 may be bad, I have never installed it before, >> only mpich1, OpenMPI and LAM. So take my mpich2 numbers with salt, Lots of >> salt. > > > I think the biggest difference in performance with various MPICH2 install > comes from differences in the 'channel' used.. I tend to make sure that I > use the 'nemesis' channel, which may or may not be the default these days. > If not, though, most people would probably want it. I think it has issues > with threading (or did ages ago?), but I seem to recall it being > considerably faster than even the 'ssm' channel. > > Sangamesh: My advice to you would be to recompile Gromacs and specify, > in the *Gromacs* compile / configure, to use the same CFLAGS you used with > MPICH2. Eg, "-O2 -m64", whatever. If you do that, I bet the times between > MPICH2 and OpenMPI will be pretty comparable for your benchmark case - > especially when run on a single processor. > I reinstalled all softwares with -O3 optimization. Following are the performance numbers for a 4 process job on a single node: MPICH2: 26 m 54 s OpenMPI: 24 m 39 s More details: $ /home/san/PERF_TEST/mpich2/bin/mpich2version MPICH2 Version: 1.0.7 MPICH2 Release date:Unknown, built on Mon Oct 13 18:02:13 IST 2008 MPICH2 Device: ch3:sock MPICH2 configure: --prefix=/home/san/PERF_TEST/mpich2 MPICH2 CC: /usr/bin/gcc -O3 -O2 MPICH2 CXX: /usr/bin/g++ -O2 MPICH2 F77: /usr/bin/gfortran -O3 -O2 MPICH2 F90: /usr/bin/gfortran -O2 $ /home/san/PERF_TEST/openmpi/bin/ompi_info Open MPI: 1.2.7 Open MPI SVN revision: r19401 Open RTE: 1.2.7 Open RTE SVN revision: r19401 OPAL: 1.2.7 OPAL SVN revision: r19401 Prefix: /home/san/PERF_TEST/openmpi Configured architecture: x86_64-unknown-linux-gnu Configured by: san Configured on: Mon Oct 13 19:10:13 IST 2008 Configure host: locuzcluster.org Built by: san Built on: Mon Oct 13 19:18:25 IST 2008 Built host: locuzcluster.org C bindings: yes C++ bindings: yes Fortran77 bindings: yes (all) Fortran90 bindings: yes Fortran90 bindings size: small C compiler: /usr/bin/gcc C compiler absolute: /usr/bin/gcc C++ compiler: /usr/bin/g++ C++ compiler absolute: /usr/bin/g++ Fortran77 compiler: /usr/bin/gfortran Fortran77 compiler abs: /usr/bin/gfortran Fortran90 compiler: /usr/bin/gfortran Fortran90 compiler abs: /usr/bin/gfortran C profiling: yes C++ profiling: yes Fortran77 profiling: yes Fortran90 profiling: yes C++ exceptions: no Thread support: posix (mpi: no, progress: no) Internal debug support: no MPI parameter check: runtime Memory profiling support: no Memory debugging support: no libltdl support: yes Heterogeneous support: yes mpirun default --prefix: no Thanks, Sangamesh > > Cheers, > - Brian > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > MPICH2 [san@locuzcluster mpich2-1.0.7]$ ./configure --help [san@locuzcluster mpich2-1.0.7]$ export CC=`which gcc` [san@locuzcluster mpich2-1.0.7]$ export CXX=`which g++` [san@locuzcluster mpich2-1.0.7]$ export F77=`which gfortarn` [san@locuzcluster mpich2-1.0.7]$ export F90=`which gfortran` [san@locuzcluster mpich2-1.0.7]$ export CFLAGS=-O3 [san@locuzcluster mpich2-1.0.7]$ export FFLAGS=-O3 [san@locuzcluster mpich2-1.0.7]$ ./configure --prefix=/home/san/PERF_TEST/mpich2 | tee config_out [san@locuzcluster mpich2-1.0.7]$ make | tee make_out OPENMPI [san@locuzcluster openmpi-1.2.7]$ export FC=`which gfortran` [san@locuzcluster openmpi-1.2.7]$ ./configure --prefix=/home/san/PERF_TEST/openmpi | tee config_out [san@locuzcluster openmpi-1.2.7]$ make | tee make_out [san@locuzcluster openmpi-1.2.7]$ make install | tee install_out FFTW $ export CC=`which gcc` $ export CXX=`which g++` $ export F77=`which gfortran` $ export CFLAGS=-O3 $ export FFLAGS=-O3 GROMACS With MPICH2 $ export CC=`which gcc` $ export CXX=`which g++` $ export F77=`which gfortran` $ export CFLAGS=
Re: [OMPI users] OPENMPI 1.2.7 & PGI compilers: configure option --disable-ptmalloc2-opt-sbrk
Hello Jeff First of all About your replay: ³ I'm surprised that we have not yet put this info in our FAQ -- I'll make a note to do so... ³ I sent support at Mailing list on date: 2008-08-21 12:43:40 Subject: [OMPI users] Memory allocation with PGI compiler About ptmalloc: I have a cluster of 16 nodes DualCPU DualCore AMD RAM 16 GB with InfiniBand CISCO HCA and switch InfiniBand. It uses Linux RH Enterprise 4 64 bit , OpenMPI 1.2.7, PGI 7.1-4 and openib-1.2-7. Hence it means that the option disable-ptmalloc2 is catastrophic in the above configuration. Can you help me about this issue in memory allocation in Openmpi 1.2.7 and PGI. I tested also with PGI 7.2 with same effects. Thans in advance On 11/10/08 14:03, "Jeff Squyres" wrote: > I'm surprised that we have not yet put this info in our FAQ -- I'll > make a note to do so... > > ptmalloc2 is a memory allocator that is only relevant for OpenFabrics > networks. If you're not using OpenFabrics, then you don't need it, > and you can safely disable ptmalloc2. FWIW, we have changed the > default behavior in the upcoming Open MPI v1.3 to include include > ptmalloc2 by default. > > > > On Oct 10, 2008, at 11:18 AM, Francesco Iannone wrote: > >> Dear openmpi users >> >> I have compiled oenmpi.1.2.7 with PGI 7.1-4 compilers with configure >> option ³--disable-ptmalloc2-opt-sbrk² , to fix a segmentation fault >> in sysMALLOC function of ³opal/mca/memory/ptmalloc2/malloc.c². >> >> Anybody knows what it means to compile with this option ? >> >> thanks >> >> >> Dr. Francesco Iannone >> Associazione EURATOM-ENEA sulla Fusione >> C.R. ENEA Frascati >> Via E. Fermi 45 >> 00044 Frascati (Roma) Italy >> phone 00-39-06-9400-5124 >> fax 00-39-06-9400-5524 >> mailto:francesco.iann...@frascati.enea.it >> http://www.afs.enea.it/iannone >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > Dr. Francesco Iannone Associazione EURATOM-ENEA sulla Fusione C.R. ENEA Frascati Via E. Fermi 45 00044 Frascati (Roma) Italy phone 00-39-06-9400-5124 fax 00-39-06-9400-5524 mailto:francesco.iann...@frascati.enea.it http://www.afs.enea.it/iannone
Re: [OMPI users] Performance: MPICH2 vs OpenMPI
For MPICH2 1.0.7, configure with --with-device=ch3:nemesis. That will use shared memory within a node unlike ch3:sock which uses TCP. Nemesis is the default in 1.1a1. Rajeev > Date: Wed, 15 Oct 2008 18:21:17 +0530 > From: "Sangamesh B" > Subject: Re: [OMPI users] Performance: MPICH2 vs OpenMPI > To: "Open MPI Users" > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > On Fri, Oct 10, 2008 at 10:40 PM, Brian Dobbins > wrote: > > > > > Hi guys, > > > > On Fri, Oct 10, 2008 at 12:57 PM, Brock Palen > wrote: > > > >> Actually I had a much differnt results, > >> > >> gromacs-3.3.1 one node dual core dual socket opt2218 > openmpi-1.2.7 > >> pgi/7.2 > >> mpich2 gcc > >> > > > >For some reason, the difference in minutes didn't come > through, it > > seems, but I would guess that if it's a medium-large > difference, then it has > > its roots in PGI7.2 vs. GCC rather than MPICH2 vs. OpenMPI. > Though, to be > > fair, I find GCC vs. PGI (for C code) is often a toss-up - > one may beat the > > other handily on one code, and then lose just as badly on another. > > > > I think my install of mpich2 may be bad, I have never > installed it before, > >> only mpich1, OpenMPI and LAM. So take my mpich2 numbers > with salt, Lots of > >> salt. > > > > > > I think the biggest difference in performance with > various MPICH2 install > > comes from differences in the 'channel' used.. I tend to > make sure that I > > use the 'nemesis' channel, which may or may not be the > default these days. > > If not, though, most people would probably want it. I > think it has issues > > with threading (or did ages ago?), but I seem to recall it being > > considerably faster than even the 'ssm' channel. > > > > Sangamesh: My advice to you would be to recompile > Gromacs and specify, > > in the *Gromacs* compile / configure, to use the same > CFLAGS you used with > > MPICH2. Eg, "-O2 -m64", whatever. If you do that, I bet > the times between > > MPICH2 and OpenMPI will be pretty comparable for your > benchmark case - > > especially when run on a single processor. > > > > I reinstalled all softwares with -O3 optimization. Following are the > performance numbers for a 4 process job on a single node: > > MPICH2: 26 m 54 s > OpenMPI: 24 m 39 s > > More details: > > $ /home/san/PERF_TEST/mpich2/bin/mpich2version > MPICH2 Version: 1.0.7 > MPICH2 Release date:Unknown, built on Mon Oct 13 18:02:13 IST 2008 > MPICH2 Device: ch3:sock > MPICH2 configure: --prefix=/home/san/PERF_TEST/mpich2 > MPICH2 CC: /usr/bin/gcc -O3 -O2 > MPICH2 CXX: /usr/bin/g++ -O2 > MPICH2 F77: /usr/bin/gfortran -O3 -O2 > MPICH2 F90: /usr/bin/gfortran -O2 > > > $ /home/san/PERF_TEST/openmpi/bin/ompi_info > Open MPI: 1.2.7 >Open MPI SVN revision: r19401 > Open RTE: 1.2.7 >Open RTE SVN revision: r19401 > OPAL: 1.2.7 >OPAL SVN revision: r19401 > Prefix: /home/san/PERF_TEST/openmpi > Configured architecture: x86_64-unknown-linux-gnu >Configured by: san >Configured on: Mon Oct 13 19:10:13 IST 2008 > Configure host: locuzcluster.org > Built by: san > Built on: Mon Oct 13 19:18:25 IST 2008 > Built host: locuzcluster.org > C bindings: yes > C++ bindings: yes > Fortran77 bindings: yes (all) > Fortran90 bindings: yes > Fortran90 bindings size: small > C compiler: /usr/bin/gcc > C compiler absolute: /usr/bin/gcc > C++ compiler: /usr/bin/g++ >C++ compiler absolute: /usr/bin/g++ > Fortran77 compiler: /usr/bin/gfortran > Fortran77 compiler abs: /usr/bin/gfortran > Fortran90 compiler: /usr/bin/gfortran > Fortran90 compiler abs: /usr/bin/gfortran > C profiling: yes >C++ profiling: yes > Fortran77 profiling: yes > Fortran90 profiling: yes > C++ exceptions: no > Thread support: posix (mpi: no, progress: no) > Internal debug support: no > MPI parameter check: runtime > Memory profiling support: no > Memory debugging support: no > libltdl support: yes >Heterogeneous support: yes > mpirun default --prefix: no > > Thanks, > Sangamesh
Re: [OMPI users] OPENMPI 1.2.7 & PGI compilers: configure option --disable-ptmalloc2-opt-sbrk
On Oct 15, 2008, at 9:35 AM, Francesco Iannone wrote: I have a cluster of 16 nodes DualCPU DualCore AMD RAM 16 GB with InfiniBand CISCO HCA and switch InfiniBand. It uses Linux RH Enterprise 4 64 bit , OpenMPI 1.2.7, PGI 7.1-4 and openib-1.2-7. Hence it means that the option —disable-ptmalloc2 is catastrophic in the above configuration. Actually, I notice that in your original message, you said "--disable- ptmalloc2-opt-sbrk", but here you said "--disable-ptmalloc2". The former is: Only trigger callbacks when sbrk is used for small allocations, rather than every call to malloc/free. (default: enabled) So it should be fine to disable; it shouldn't affect overall MPI performance too much. The latter disables ptmalloc2 entirely (and you'll likely get lower benchmark bandwidth for large messages). I'm unaware of either of these options leading to problems with the PGI compiler suite; I have tested OMPI v1.2.x with several versions of the PGI compiler without problem (although my latest version is PGI 7.1-4). -- Jeff Squyres Cisco Systems
