[OMPI users] Help with HPL and MPI
Hello all, I am trying to configure HPL on a beowulf cluster that I have put together for a senior project at Weber State University, and I am having a little bit of trouble. First of all, about the cluster: 4-node diskless cluster Fedora Core 6 - 64 bit version Intel Pentium D dual core processors MSI 965 motherboards Right now I have one node doing a net boot with syslinux and would like to do a benchmark before I put the rest of the nodes together, so I can have a performance comparison with subsequent nodes. I have installed the following packages on my system for HPL: openmpi-1.1-7.fc6.x86_64.rpm openmpi-devel-1.1-7.fc6.x86_64.rpm openmpi-libs-1.1-7.fc6.x86_64.rpm lapack-3.1.0-4.fc6.x86_64.rpm Blas-3.1.0-4.fc6.x86_64.rpm atlas-4.6.0-11.fc6.x86_64.rpm cblas.tgz hpl.tgz I may have installed more packages than necessary but I didn't think it would hurt. Everything has installed successfully but I can't get the makefile. down. I simply just don't understand enough of it to build it correctly. I just keep getting 'Make.inc' errors. The Makefile that I have attempted is below, called Make.Beowulf. I just used a generic makefile from the setups directory and attempted to supply some paths to the libraries but to no avail. I have tried to find documentation explaining more clearly how everything should be setup but nothing in lay-man terms, hence the errors. A few questions: What should my arch be? Does that even matter? Does it have to be x86_64? I realize I have to supply paths to the BLAS and MPI headers and libraries but exactly which libraries and header files? The compiler I am using is mpicc which is just linked to gcc, but shouldn't that compiler supply the links to the correct libraries and header files? The MPlib parameter points to libmpich.a so I installed mpich2 but that didn't give me a libmpich.a directory so what should I use there? Also, I am not using an network file systems so am I correct in assuming that all of the libraries need to be on each of the nodes? If so, I need to know exactly where to put them, and again, I believe they would need to be put into the exact same location, so the problem is, which libraries and header files exactly? (as to save precious RAM on each of the nodes). I realize I may be asking a lot but the end of the semester is just around the corner. I appreciate any help that you may give me ahead of time. Thanks. Stephen Jenkins sn...@comcast.net Make.Beowulf SHELL = /bin/sh # CD = cd CP = cp LN_S = ln -s MKDIR = mkdir RM = /bin/rm -f TOUCH = touch # - Platform identifier ARCH = Linux_x86_64 # - HPL Directory Structure / HPL library -- TOPdir = $(HOME)/hpl INCdir = $(TOPdir)/include BINdir = $(TOPdir)/bin/$(ARCH) LIBdir = $(TOPdir)/lib/$(ARCH) # HPLlib = $(LIBdir)/libhpl.a # - Message Passing library (MPI) -- MPdir = /usr/include/openmpi MPinc = -I$/usr/include/include MPlib = $(MPdir)/lib/libmpich.a # - Linear Algebra library (BLAS or VSIPL) - LAdir = $(HOME)/netlib/ARCHIVES/Linux_PII LAinc = LAlib = $(LAdir)/libcblas.a $(LAdir)/libatlas.a # - F77 / C interface -- F2CDEFS = # - HPL includes / libraries / specifics --- HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc) HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib) # - Compile time options --- HPL_OPTS = -DHPL_CALL_CBLAS # -- HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES) # - Compilers / linkers - Optimization flags --- CC = /usr/bin/gcc CCNOOPT = $(HPL_DEFS) CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops # On some platforms, it is necessary to use the Fortran linker to find # the Fortran internals used in the BLAS library. LINKER = /usr/bin/g77 LINKFLAGS = $(CCFLAGS) # ARCHIVER = ar ARFLAGS = r RANLIB = echo
Re: [OMPI users] OpenMPI run with the SGE launcher, orte PE calrification
Thanks Goetz, we are able to submit OpenMPI jobs using SGE. But still when we try to submit interactive job we are getting the following error, $ qrsh -pe mpich 4 Rocks Compute Node Rocks 4.2.1 (Cydonia) Profile built 14:09 02-Apr-2007 Kickstarted 22:16 02-Apr-2007 -bash: ulimit: max locked memory: cannot modify limit: Operation not permitted we are setting the ulimit in "/etc/init.d/sshd" file, but not sure why we are getting this ulimit error only in interactive jobs. How do you set ulimit?? and how much?? pls advice. thanks, -bala- --- Götz Waschk wrote: > On 3/28/07, Bala wrote: > > % qconf -sp orte > > > > start_proc_args /bin/true > > stop_proc_args/bin/true > > just want to know anybody successfully running in > SGE > > using this PE?? > > Hi, > > yes, I have a working installation of openmpi 1.2 > with SGE 6.0u9. > > > from my mpich PE I can see start/stop arguments as > > show below > > start_proc_args /opt/gridengine/mpi/startmpi.sh > > -catch_rsh > > $pe_hostfile > > stop_proc_args/opt/gridengine/mpi/stopmpi.sh > > Take a look at these scripts, they don't do anything > fancy, they just > prepare the machinefile and the rsh wrapper for > mpich. Openmpi does > the right thing by itself by looking at SGE's > environment variables. > > Regards, Götz Waschk - DESY Zeuthen > > -- > AL I:40: Do what thou wilt shall be the whole of the > Law. > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > Looking for earth-friendly autos? Browse Top Cars by "Green Rating" at Yahoo! Autos' Green Center. http://autos.yahoo.com/green_center/
Re: [OMPI users] OpenMPI run with the SGE launcher, orte PE calrification
On 4/10/07, Bala wrote: But still when we try to submit interactive job we are getting the following error, $ qrsh -pe mpich 4 Rocks Compute Node Rocks 4.2.1 (Cydonia) Profile built 14:09 02-Apr-2007 Kickstarted 22:16 02-Apr-2007 -bash: ulimit: max locked memory: cannot modify limit: Operation not permitted Hi Bala, it seems you set the limit in one of the bash profile scripts, be it ~/.profile, /etc/profile or /etc/profile.d/*. How do you set ulimit?? and how much?? pls advice. I don't set this limit at all, SGE doesn't touch the default value of 32k. There are several ways to set the limit. It seems you are hitting the hard limit, if you want to set a higher value, you have to modify /etc/security/limits.conf as defined in the comments. This is part of pam, so you have to make sure your ssh session is using pam. Regards, Götz Waschk -- AL I:40: Do what thou wilt shall be the whole of the Law.
Re: [OMPI users] OpenMPI run with the SGE launcher, orte PE calrification
If you are setting the limits for locked memory for InfiniBand, see these FAQ entries: http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-more On Apr 10, 2007, at 6:36 AM, Götz Waschk wrote: On 4/10/07, Bala wrote: But still when we try to submit interactive job we are getting the following error, $ qrsh -pe mpich 4 Rocks Compute Node Rocks 4.2.1 (Cydonia) Profile built 14:09 02-Apr-2007 Kickstarted 22:16 02-Apr-2007 -bash: ulimit: max locked memory: cannot modify limit: Operation not permitted Hi Bala, it seems you set the limit in one of the bash profile scripts, be it ~/.profile, /etc/profile or /etc/profile.d/*. How do you set ulimit?? and how much?? pls advice. I don't set this limit at all, SGE doesn't touch the default value of 32k. There are several ways to set the limit. It seems you are hitting the hard limit, if you want to set a higher value, you have to modify /etc/security/limits.conf as defined in the comments. This is part of pam, so you have to make sure your ssh session is using pam. Regards, Götz Waschk -- AL I:40: Do what thou wilt shall be the whole of the Law. ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems
Re: [OMPI users] Help with HPL and MPI
I remember trying to build HPL my first time as a student :-) Its much simpler than you think It also is a great way to learn about linking and make files. (Well in my opinion).See my comments below, mostly quickly looking is you should change g77 and gcc to mpicc. Yes mpicc calls gcc you can see by runningmpicc -showmeThus you can ommit the MPdir, MPinc, and MPlib (No libmpich.a !)You can see one I use to ACML on our cluster, note we use PGI compilers. Make.OPT_OMPI Description: Binary data Brock PalenCenter for Advanced Computingbro...@umich.edu(734)936-1985 On Apr 10, 2007, at 12:10 AM, sn...@comcast.net wrote: Hello all, I am trying to configure HPL on a beowulf cluster that I have put together for a senior project at Weber State University, and I am having a little bit of trouble. First of all, about the cluster: 4-node diskless cluster Fedora Core 6 - 64 bit version Intel Pentium D dual core processors MSI 965 motherboards Right now I have one node doing a net boot with syslinux and would like to do a benchmark before I put the rest of the nodes together, so I can have a performance comparison with subsequent nodes. I have installed the following packages on my system for HPL: openmpi-1.1-7.fc6.x86_64.rpm openmpi-devel-1.1-7.fc6.x86_64.rpm openmpi-libs-1.1-7.fc6.x86_64.rpm lapack-3.1.0-4.fc6.x86_64.rpm Blas-3.1.0-4.fc6.x86_64.rpm atlas-4.6.0-11.fc6.x86_64.rpm cblas.tgz hpl.tgz I may have installed more packages than necessary but I didn't think it would hurt. Everything has installed successfully but I can't get the makefile. down. I simply just don't understand enough of it to build it correctly. I just keep getting 'Make.inc' errors. The Makefile that I have attempted is below, called Make.Beowulf. I just used a generic makefile from the setups directory and attempted to supply some paths to the libraries but to no avail. I have tried to find documentation explaining more clearly how everything should be setup but nothing in lay-man terms, hence the errors. A few questions: What should my arch be? Does that even matter? Does it have to be x86_64? I realize I have to supply paths to the BLAS and MPI headers and libraries but exactly which libraries and header files? The compiler I am using is mpicc which is just linked to gcc, but shouldn't that compiler supply the links to the correct libraries and header files? The MPlib parameter points to libmpich.a so I installed mpich2 but that didn't give me a libmpich.a directory so what should I use there? Also, I am not using an network file systems so am I correct in assuming that all of the libraries need to be on each of the nodes? If so, I need to know exactly where to put them, and again, I believe they would need to be put into the exact same location, so the problem is, which libraries and header files exactly? (as to save precious RAM on each of the nodes). I realize I may be asking a lot but the end of the semester is just around the corner. I appreciate any help that you may give me ahead of time. Thanks. Stephen Jenkins sn...@comcast.net Make.Beowulf SHELL = /bin/sh#CD = cdCP = cpLN_S = ln -sMKDIR = mkdirRM = /bin/rm -fTOUCH = touch# - Platform identifier ARCH = Linux_x86_64# - HPL Directory Structure / HPL library --TOPdir = $(HOME)/hplINCdir = $(TOPdir)/includeBINdir = $(TOPdir)/bin/$(ARCH)LIBdir = $(TOPdir)/lib/$(ARCH)#HPLlib = $(LIBdir)/libhpl.a # - Message Passing library (MPI) --MPdir = /usr/include/openmpiMPinc = -I$/usr/include/includeMPlib = $(MPdir)/lib/libmpich.a# - Linear Algebra library (BLAS or VSIPL) -LAdir = $(HOME)/netlib/ARCHIVES/Linux_PIILAinc =LAlib = $(LAdir)/libcblas.a $(LAdir)/libatlas.a# - F77 / C interface --F2CDEFS =# - HPL includes / libraries / specifics ---HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)HPL_LIBS = $(HPLlib) $(LAlib) $(MPlib)# - Compile time options ---HPL_OPTS = -DHPL_CALL_CBLAS# --HPL_DEFS = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)# - Compilers / linkers - Optimization flags ---CC = /usr/bin/gccChange this to mpiccCCNOOPT = $(HPL_DEFS)CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops# On some platforms, it is necessary to use the Fortran linker to find# the Fortran internals used in the BLAS library.LINKER = /usr/bin/g77Why? Also make this mpiccLINKFLAGS = $(CCFLAGS)#ARCHIVER = arARFLAGS = rRANLIB = echo___users mailing listus...@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] -prefix not working
Hi Jeff
Comments after your text
Jeff Squyres wrote 2/5/07:
Greetings Joe.
What we did was to make 2 sets of environment variables that you can
use to modify OMPI's internal path behavior:
1. OPAL_DESTDIR: If you want to use staged builds (a la RPM builds),
you can set the environment variable OPAL_DESTDIR. For example:
./configure --prefix=/opt/ompi
make install DESTDIR=/tmp
# at this point, OMPI is installed into /tmp/opt/ompi
setenv OPAL_DESTDIR /tmp
# put /tmp/opt/ompi in your path, then run ompi_info
ompi_info
Hence, OPAL_DESTDIR simply prefixes every internal path with the
destdir.
2. OPAL_PREFIX: If you want to wholly override where the OMPI
installation is, set OPAL_PREFIX to the new prefix value (quite
similar to the LAMHOME environment variable, but slightly better).
For example:
./configure --prefix=/opt/ompi
make install
mv /opt/ompi /home/ompi
setenv OPAL_PREFIX /home/ompi
ompi_info
This will re-base all of OMPI's internal directories to key off the
new prefix value. Similarly, OPAL_* environment variables exist for
all other GNU-standard directories (OPAL_BINDIR,
OPAL_INCLUEDIR, ...etc.). Note that the default values of these
variables are their GNU counterparts, so OMPI_BINDIR defaults to $
{prefix}/bin -- which requires resolving ${prefix} first (i.e.,
OPAL_PREFIX). Hence, setting OPAL_PREFIX will effectively change all
the others. We anticipate that most people will likely only ever use
OPAL_PREFIX.
I finally got a chance to try V1.2. I am having the same issue:
I installed with:
./configure --prefix=/usr/local/openmpi-1.2-1
make prefix=/usr/local/openmpi-1.2-1
make install prefix=/usr/local/openmpi-1.2-1
I can run with:
em64t6 <155> /usr/local/openmpi-1.2-1/bin/mpirun -n 1 --host em64t6 hostname
em64t6
em64t6 <156> cp -r /usr/local/openmpi-1.2-1 /tmp
em64t6 <157> /tmp/openmpi-1.2-1/bin/mpirun -n 1 --host em64t6 hostname
em64t6
Then I move the directory:
[root@em64t6 openmpi-1.2]# cd /usr/local/
[root@em64t6 local]# mv openmpi-1.2-1/ openmpiHIDE
Then I try to run:
em64t6 <158> /tmp/openmpi-1.2-1/bin/mpirun -n 1 --host em64t6 hostname
/tmp/openmpi-1.2-1/bin/mpirun: error while loading shared libraries:
libopen-rte.so.0: cannot open shared object file: No such file or directory
If I set the env:
em64t6 <160> setenv OPAL_PREFIX /tmp/openmpi-1.2-1
em64t6 <164> setenv OPAL_DESTDIR /tmp/openmpi-1.2-1/
em64t6 <166> set path=( /tmp/openmpi-1.2-1/bin $path)
em64t6 <168> setenv LD_LIBRARY_PATH /tmp/openmpi-1.2-1/lib
The error becomes:
em64t6 <169> /tmp/openmpi-1.2-1/bin/mpirun -n 1 --host em64t6 hostname
[em64t6:30103] [NO-NAME] ORTE_ERROR_LOG: Not found in file
runtime/orte_init_stage1.c at line 214
--
Sorry! You were supposed to get help about:
orte_init:startup:internal-failure
from the file:
help-orte-runtime
But I couldn't find any file matching that name. Sorry!
--
[em64t6:30103] [NO-NAME] ORTE_ERROR_LOG: Not found in file
runtime/orte_system_init.c at line 42
[em64t6:30103] [NO-NAME] ORTE_ERROR_LOG: Not found in file
runtime/orte_init.c at line 52
--
Sorry! You were supposed to get help about:
orterun:init-failure
from the file:
help-orterun.txt
But I couldn't find any file matching that name. Sorry!
--
The output from ompi_info is:
em64t6 <170> ompi_info
Open MPI: 1.2
Open MPI SVN revision: r14027
Open RTE: 1.2
Open RTE SVN revision: r14027
OPAL: 1.2
OPAL SVN revision: r14027
Prefix: /usr/local/openmpi-1.2-1
Configured architecture: x86_64-unknown-linux-gnu
Configured by: root
Configured on: Tue Apr 10 09:39:28 PDT 2007
Configure host: em64t6
Built by: root
Built on: Tue Apr 10 09:51:27 PDT 2007
Built host: em64t6
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: no
Fortran90 bindings size: na
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: g77
Fortran77 compiler abs: /usr/bin/g77
Fortran90 compiler: none
Fortran90 compiler abs: none
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: no
C++ exceptions: no
Thread support: posix (mpi: no, progress: no)
Internal debug support: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: yes
mpirun default --prefix: no
MCA backtr
Re: [OMPI users] OpenMPI for Windows?
On Mar 30, 2007, at 4:23 PM, rohit_si...@logitech.com wrote: I'm somewhat new to OpenMPI, but I'm currently evaluating it as a communications mechanism between Windows and Unix servers. I noticed that under your FAQs ( http://www.open-mpi.org/faq/?category=supported-systems), it says: There are plans to support Microsoft Windows in the not-distant future. When is the not-distant future? Is it in scope for this year? Will the Windows support require a Unix emulation layer like Cygwin? I do apologize if the information I'm requesting here is confidential in nature. The Windows support is planned, but we don't have a definite time table. Much of the work has been done, although there are still pieces missing. I believe remote launching is one of those pieces. The Windows support will be native and will not require Cygwin (which is part of what is taking us so long -- it's a bit project to move a UNIX-based code like Open MPI to Windows without sacrificing too much in maintainability and performance. Hope this helps, Brian
