[OMPI users] OpenMPI-1.2.8 issues on 64-bit Itanium platforms (IA64)
Dear Users, Support Engineers, I am using OpenMPI-1.2.8 on two sets of machines. The first set has three 64-bit Itanium machines and the second set has three 32-bit i386 machines. The machines in the first set are multicores. All the machines are connected by 1 Gigabit Ethernet. My complaint is I am observing large communication times between the machines, i.e., inter-node communications. My comparisions are with MPICH-1.2.7p1. I am attaching communication times for simple Ping experiments. The Ping application is attached in the file 'MPI_ping.c'. However, my real application is parallel matrix-matrix application where I am observing communication times some five times more than the computation times. On both the sets of machines, OpenMPI is installed using the simplest configure command, which is "configure --prefix=<...>". Could you please tell me if I should provide any special installation options for efficient communications? All the information related to the tests is present in the ZIP file attached. The platform information of the machines obtained using "uname -a" is printed below: ia64$ uname -a Linux rosebud08 2.6.9-42.EL #1 SMP Wed Jul 12 23:25:09 EDT 2006 ia64 ia64 ia64 GNU/Linux i386$ uname -a Linux rosebud04 2.6.9-42.0.2.ELsmp #1 SMP Wed Aug 23 00:17:26 CDT 2006 i686 i686 i386 GNU/Linux The compiler used on all these machines is Intel 'icc' 9.1.046. The CPU information for the Itanium machines is attached in the text file 'cpuinfo_ia64.txt'. The installation information for the machines (ia64, i386) are attached in the files 'config_ia64.log' and 'config_i386.log' respectively. The output from the ompi_info command for the machines (ia64, i386) are attached in the files 'ompi_info_ia64.log' and 'ompi_info_i386.log' respectively. The experiments results are shown at the end of the mail (also in the text file eresults.txt). I am also presenting below results using MPICH-1.2.7p1, whose inter-node communication times between two ia64 nodes are faster. As you can see, OpenMPI communications are expensive. My question is are there any special installation options to use for ia64 platforms? I would be using a mix of ia64 and i386 platforms in the future. Are there any issues that need to be taken care of? Thanking you for your cooperation. Best regards Ravi Reddy School of Computer Science and Informatics University College Dublin * ** ** ** WARNING: This email contains an attachment of a very suspicious type. ** ** You are urged NOT to open this attachment unless you are absolutely ** ** sure it is legitimate. Opening this attachment may cause irreparable ** ** damage to your computer and your files. If you have any questions ** ** about the validity of this message, PLEASE SEEK HELP BEFORE OPENING IT. ** ** ** ** This warning was added by the IU Computer Science Dept. mail scanner. ** * <>
[OMPI users] Fwd: OpenMPI-1.2.8 issues on 64-bit Itanium platforms (IA64)
Dear Users, Support Engineers,I am sorry for this extra post. But it has more accurate results in the file 'eresults.txt'. I am using OpenMPI-1.2.8 on two sets of machines. The first set has three 64-bit Itanium machines and the second set has three 32-bit i386 machines. The machines in the first set are multicores. All the machines are connected by 1 Gigabit Ethernet. My complaint is I am observing large communication times between the machines, i.e., inter-node communications. My comparisions are with MPICH-1.2.7p1. I am attaching communication times for simple Ping experiments. The Ping application is attached in the file 'MPI_ping.c'. However, my real application is parallel matrix-matrix application where I am observing communication times some five times more than the computation times. On both the sets of machines, OpenMPI is installed using the simplest configure command, which is "configure --prefix=<...>". Could you please tell me if I should provide any special installation options for efficient communications? All the information related to the tests is present in the ZIP file attached. The platform information of the machines obtained using "uname -a" is printed below:ia64$ uname -aLinux rosebud08 2.6.9-42.EL #1 SMP Wed Jul 12 23:25:09 EDT 2006 ia64 ia64 ia64 GNU/Linuxi386$ uname -aLinux rosebud04 2.6.9-42.0.2.ELsmp #1 SMP Wed Aug 23 00:17:26 CDT 2006 i686 i686 i386 GNU/Linux The compiler used on all these machines is Intel 'icc' 9.1.046. The CPU information for the Itanium machines is attached in the text file 'cpuinfo_ia64.txt'. The installation information for the machines (ia64, i386) are attached in the files 'config_ia64.log' and 'config_i386.log' respectively. The output from the ompi_info command for the machines (ia64, i386) are attached in the files 'ompi_info_ia64.log' and 'ompi_info_i386.log' respectively. The experiments results are shown at the end of the mail (also in the text file eresults.txt). I am also presenting below results using MPICH-1.2.7p1, whose inter-node communication times between two ia64 nodes are faster. As you can see, OpenMPI communications are expensive. My question is are there any special installation options to use for ia64 platforms? I would be using a mix of ia64 and i386 platforms in the future. Are there any issues that need to be taken care of? Thanking you for your cooperation. Best regardsRavi ReddySchool of Computer Science and InformaticsUniversity College Dublin * ** ** ** WARNING: This email contains an attachment of a very suspicious type. ** ** You are urged NOT to open this attachment unless you are absolutely ** ** sure it is legitimate. Opening this attachment may cause irreparable ** ** damage to your computer and your files. If you have any questions ** ** about the validity of this message, PLEASE SEEK HELP BEFORE OPENING IT. ** ** ** ** This warning was added by the IU Computer Science Dept. mail scanner. ** * <>
[OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9; problems on heterogeneous cluster
Hi,
I am facing problems running OpenMPI-1.0.1 on a heterogeneous cluster.
I have a Linux machine and a SunOS machine in this cluster.
linux$ uname -a
Linux pg1cluster01 2.6.8-1.521smp #1 SMP Mon Aug 16 09:25:06 EDT 2004
i686 i686 i386 GNU/Linux
OpenMPI-1.0.1 is installed uisng
./configure --prefix=...
make all install
sunos$ uname -a
SunOS csultra01 5.9 Generic_112233-10 sun4u sparc SUNW,Ultra-5_10
OpenMPI-1.0.1 is installed uisng
./configure --prefix=...
make all install
I use ssh. Both nodes are accessible without prompts for password.
I use the following simple application:
#include
int main(int argc, char** argv)
{
int rc, me;
char pname[MPI_MAX_PROCESSOR_NAME];
int plen;
MPI_Init(
&argc,
&argv
);
rc = MPI_Comm_rank(
MPI_COMM_WORLD,
&me
);
if (rc != MPI_SUCCESS)
{
return rc;
}
MPI_Get_processor_name(
pname,
&plen
);
printf("%s:Hello world from %d\n", pname, me);
MPI_Finalize();
return 0;
}
It is compiled as follows:
linux$ mpicc -o mpiinit_linux mpiinit.c
sunos$ mpicc -o mpiinit_sunos mpiinit.c
My hosts file is
hosts.txt
-
pg1cluster01 slots=2
csultra01 slots=1
My app file is
mpiinit_appfile
---
-np 2 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_linux
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos
$ mpirun --hostfile hosts.txt --app mpiinit_appfile
ld.so.1: /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos:
fatal: relocation error: file
/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib/libmca_common_sm.so.0:
symbol nanosleep: referenced symbol not found
ld.so.1: /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos:
fatal: relocation error: file
/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib/libmca_common_sm.so.0:
symbol nanosleep: referenced symbol not found
I have fixed this by compiling with "-lrt" option to the linker.
sunos$ mpicc -o mpiinit_sunos mpiinit.c -lrt
However when I run this again, I get the error:
$ mpirun --hostfile hosts.txt --app mpiinit_appfile
[pg1cluster01:19858] ERROR: A daemon on node csultra01 failed to start
as expected.
[pg1cluster01:19858] ERROR: There may be more information available from
[pg1cluster01:19858] ERROR: the remote shell (see above).
[pg1cluster01:19858] ERROR: The daemon exited unexpectedly with status 255.
2 processes killed (possibly by Open MPI)
Sometimes I get the error.
$ mpirun --hostfile hosts.txt --app mpiinit_appfile
[csultra01:06256] mca_common_sm_mmap_init: ftruncate failed with errno=28
[csultra01:06256] mca_mpool_sm_init: unable to create shared memory mapping
--
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
PML add procs failed
--> Returned value -2 instead of OMPI_SUCCESS
--
*** An error occurred in MPI_Init
*** before MPI was initialized
*** MPI_ERRORS_ARE_FATAL (goodbye)
Please let me know the resolution of this problem. Please let me know if
you need more details.
Regards,
Ravi.
Re: [OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9; problems on heterogeneous cluster
Hi Brian, Thank you for your help. I have attached all the files you have asked for in a tar file. Please find attached the 'config.log' and 'libmpi.la' for my Solaris installation. The output from 'mpicc -showme' is sunos$ mpicc -showme gcc -I/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/include -I/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/include/openmpi/ompi -L/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib -lmpi -lorte -lopal -lnsl -lsocket -lthread -laio -lm -lnsl -lsocket -lthread -ldl There are serious issues when running on just solaris machines. I am using the host file and app file shown below. Both the machines are SunOS and are similar. hosts.txt - csultra01 slots=1 csultra02 slots=1 mpiinit_appfile --- -np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos -np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos Running mpirun without -d option hangs. csultra01$ mpirun --hostfile hosts.txt --app mpiinit_appfile hangs Running mpirun with -d option dumps core with output in the file "mpirun_output_d_option.txt", which is attached. The core is also attached. Running just on only one host is also not working. The output from mpirun using "-d" option for this scenario is attached in file "mpirun_output_d_option_one_host.txt". I have also attached the list of packages installed on my solaris machine in "pkginfo.txt" I hope these will help you to resolve the issue. Regards, Ravi. - Original Message - From: Brian Barrett List-Post: users@lists.open-mpi.org Date: Friday, March 10, 2006 7:09 pm Subject: Re: [OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9; problems on heterogeneous cluster To: Open MPI Users > On Mar 10, 2006, at 12:09 AM, Ravi Manumachu wrote: > > > I am facing problems running OpenMPI-1.0.1 on a heterogeneous > cluster.> > > I have a Linux machine and a SunOS machine in this cluster. > > > > linux$ uname -a > > Linux pg1cluster01 2.6.8-1.521smp #1 SMP Mon Aug 16 09:25:06 EDT > 2004> i686 i686 i386 GNU/Linux > > > > sunos$ uname -a > > SunOS csultra01 5.9 Generic_112233-10 sun4u sparc SUNW,Ultra-5_10 > > Unfortunately, this will not work with Open MPI at present. Open > MPI > 1.0.x does not have any support for running across platforms with > different endianness. Open MPI 1.1.x has much better support for > such situations, but is far from complete, as the MPI datatype > engine > does not properly fix up endian issues. We're working on the > issue, > but can not give a timetable for completion. > > Also note that (while not a problem here) Open MPI also does not > support running in a mixed 32 bit / 64 bit environment. All > processes must be 32 or 64 bit, but not a mix. > > > $ mpirun --hostfile hosts.txt --app mpiinit_appfile > > ld.so.1: /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/ > > mpiinit_sunos: > > fatal: relocation error: file > > /home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib/ > > libmca_common_sm.so.0: > > symbol nanosleep: referenced symbol not found > > ld.so.1: /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/ > > mpiinit_sunos: > > fatal: relocation error: file > > /home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib/ > > libmca_common_sm.so.0: > > symbol nanosleep: referenced symbol not found > > > > I have fixed this by compiling with "-lrt" option to the linker. > > You shouldn't have to do this... Could you send me the config.log > file configure for Open MPI, the installed $prefix/lib/libmpi.la > file, and the output of mpicc -showme? > > > sunos$ mpicc -o mpiinit_sunos mpiinit.c -lrt > > > > However when I run this again, I get the error: > > > > $ mpirun --hostfile hosts.txt --app mpiinit_appfile > > [pg1cluster01:19858] ERROR: A daemon on node csultra01 failed to > start> as expected. > > [pg1cluster01:19858] ERROR: There may be more information > available > > from > > [pg1cluster01:19858] ERROR: the remote shell (see above). > > [pg1cluster01:19858] ERROR: The daemon exited unexpectedly with > > status 255. > > 2 processes killed (possibly by Open MPI) > > Both of these are quite unexpected. It looks like there is > something > wrong with your Solaris build. Can you run on *just* the Solaris > machine? We only have limited resources for testing on Solaris, > but > have not run into this issue before. What happens if you run > mpirun > on just the Solaris machine with the -d option to mpirun? > > > Sometimes I get the error. > > > &
Re: [OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9; problems on heterogeneous cluster
Hi Brian, Thank you for your help. I have attached all the files you have asked for in a tar file. Please find attached the 'config.log' and 'libmpi.la' for my Solaris installation. The output from 'mpicc -showme' is sunos$ mpicc -showme gcc -I/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/include -I/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS- 5.9/include/openmpi/ompi-L/home/cs/manredd/OpenMPI/openmpi- 1.0.1/OpenMPI-SunOS-5.9/lib -lmpi -lorte -lopal -lnsl -lsocket -lthread -laio -lm -lnsl -lsocket - lthread -ldl There are serious issues when running on just solaris machines. I am using the host file and app file shown below. Both the machines are SunOS and are similar. hosts.txt - csultra01 slots=1 csultra02 slots=1 mpiinit_appfile --- -np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos -np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos Running mpirun without -d option hangs. csultra01$ mpirun --hostfile hosts.txt --app mpiinit_appfile hangs Running mpirun with -d option dumps core with output in the file "mpirun_output_d_option.txt", which is attached. The core is also attached. Running just on only one host is also not working. The output from mpirun using "-d" option for this scenario is attached in file "mpirun_output_d_option_one_host.txt". I have also attached the list of packages installed on my solaris machine in "pkginfo.txt" I hope these will help you to resolve the issue. Regards, Ravi. > - Original Message - > From: Brian Barrett > Date: Friday, March 10, 2006 7:09 pm > Subject: Re: [OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9; > problems on heterogeneous cluster > To: Open MPI Users > > > On Mar 10, 2006, at 12:09 AM, Ravi Manumachu wrote: > > > > > I am facing problems running OpenMPI-1.0.1 on a heterogeneous > > cluster.> > > > I have a Linux machine and a SunOS machine in this cluster. > > > > > > linux$ uname -a > > > Linux pg1cluster01 2.6.8-1.521smp #1 SMP Mon Aug 16 09:25:06 > EDT > > 2004> i686 i686 i386 GNU/Linux > > > > > > sunos$ uname -a > > > SunOS csultra01 5.9 Generic_112233-10 sun4u sparc SUNW,Ultra-5_10 > > > > Unfortunately, this will not work with Open MPI at present. Open > > MPI > > 1.0.x does not have any support for running across platforms with > > > different endianness. Open MPI 1.1.x has much better support for > > > such situations, but is far from complete, as the MPI datatype > > engine > > does not properly fix up endian issues. We're working on the > > issue, > > but can not give a timetable for completion. > > > > Also note that (while not a problem here) Open MPI also does not > > support running in a mixed 32 bit / 64 bit environment. All > > processes must be 32 or 64 bit, but not a mix. > > > > > $ mpirun --hostfile hosts.txt --app mpiinit_appfile > > > ld.so.1: /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/ > > > mpiinit_sunos: > > > fatal: relocation error: file > > > /home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib/ > > > libmca_common_sm.so.0: > > > symbol nanosleep: referenced symbol not found > > > ld.so.1: /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/ > > > mpiinit_sunos: > > > fatal: relocation error: file > > > /home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/lib/ > > > libmca_common_sm.so.0: > > > symbol nanosleep: referenced symbol not found > > > > > > I have fixed this by compiling with "-lrt" option to the linker. > > > > You shouldn't have to do this... Could you send me the > config.log > > file configure for Open MPI, the installed $prefix/lib/libmpi.la > > file, and the output of mpicc -showme? > > > > > sunos$ mpicc -o mpiinit_sunos mpiinit.c -lrt > > > > > > However when I run this again, I get the error: > > > > > > $ mpirun --hostfile hosts.txt --app mpiinit_appfile > > > [pg1cluster01:19858] ERROR: A daemon on node csultra01 failed > to > > start> as expected. > > > [pg1cluster01:19858] ERROR: There may be more information > > available > > > from > > > [pg1cluster01:19858] ERROR: the remote shell (see above). > > > [pg1cluster01:19858] ERROR: The daemon exited unexpectedly with > > > > status 255. > > > 2 processes killed (possibly by Open MPI) > > > > Both of these are quite unexpected. It
Re: [OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9; problems on heterogeneous cluster
Hi Brian,
I have installed OpenMPI-1.1a1r9260 on my SunOS machines. It has solved
the problems. However there is one more issue that I found in my testing
and that I failed to report. This concerns Linux machines too.
My host file is
hosts.txt
-
csultra06
csultra02
csultra05
csultra08
My app file is
mpiinit_appfile
---
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
-np 1 /home/cs/manredd/OpenMPI/openmpi-1.1a1r9260/MPITESTS/mpiinit
My application program is
mpiinit.c
-
#include
int main(int argc, char** argv)
{
int rc, me;
char pname[MPI_MAX_PROCESSOR_NAME];
int plen;
MPI_Init(
&argc,
&argv
);
rc = MPI_Comm_rank(
MPI_COMM_WORLD,
&me
);
if (rc != MPI_SUCCESS)
{
return rc;
}
MPI_Get_processor_name(
pname,
&plen
);
printf("%s:Hello world from %d\n", pname, me);
MPI_Finalize();
return 0;
}
Compilation is successful
csultra06$ mpicc -o mpiinit mpiinit.c
However mpirun prints just 6 statements instead of 8.
csultra06$ mpirun --hostfile hosts.txt --app mpiinit_appfile
csultra02:Hello world from 5
csultra06:Hello world from 0
csultra06:Hello world from 4
csultra02:Hello world from 1
csultra08:Hello world from 3
csultra05:Hello world from 2
The following two more statements are not printed.
csultra05:Hello world from 6
csultra08:Hello world from 7
This behavior I observed on my Linux cluster too.
I have attached the log for "-d" option for your debugging purposes.
Regards,
Ravi.
- Original Message -
From: Brian Barrett
List-Post: users@lists.open-mpi.org
Date: Monday, March 13, 2006 7:56 pm
Subject: Re: [OMPI users] problems with OpenMPI-1.0.1 on SunOS 5.9;
problems on heterogeneous cluster
To: Open MPI Users
> Hi Ravi -
>
> With the help of another Open MPI user, I spent the weekend finding
> a
> couple of issues with Open MPI on Solaris. I believe you are
> running
> into the same problems. We're in the process of certifying the
> changes for release as part of 1.0.2, but it's Monday morning and
> the
> release manager hasn't gotten them into the release branch just
> yet.
> Could you give the nightly tarball from our development trunk a try
>
> and let us know if it solves your problems on Solaris? You
> probably
> want last night's 1.1a1r9260 release.
>
> http://www.open-mpi.org/nightly/trunk/
>
> Thanks,
>
> Brian
>
>
> On Mar 12, 2006, at 11:23 PM, Ravi Manumachu wrote:
>
> >
> > Hi Brian,
> >
> > Thank you for your help. I have attached all the files you have
> asked> for in a tar file.
> >
> > Please find attached the 'config.log' and 'libmpi.la' for my
> Solaris> installation.
> >
> > The output from 'mpicc -showme' is
> >
> > sunos$ mpicc -showme
> > gcc -I/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-5.9/
> > include
> > -I/home/cs/manredd/OpenMPI/openmpi-1.0.1/OpenMPI-SunOS-
> > 5.9/include/openmpi/ompi-L/home/cs/manredd/OpenMPI/openmpi-
> > 1.0.1/OpenMPI-SunOS-5.9/lib -lmpi
> > -lorte -lopal -lnsl -lsocket -lthread -laio -lm -lnsl -lsocket -
> > lthread -ldl
> >
> > There are serious issues when running on just solaris machines.
> >
> > I am using the host file and app file shown below. Both the
> > machines are
> > SunOS and are similar.
> >
> > hosts.txt
> > -
> > csultra01 slots=1
> > csultra02 slots=1
> >
> > mpiinit_appfile
> > ---
> > -np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos
> > -np 1 /home/cs/manredd/OpenMPI/openmpi-1.0.1/MPITESTS/mpiinit_sunos
> >
> > Running mpirun without -d option hangs.
> >
> > csultra01$ mpirun --hostfile hosts.txt --app mpiinit_appfile
> > hangs
> >
> > Running mpirun with -d option dumps core with output in the file
> > "mpirun_output_d_option.txt", which is attached. The core is also
> > attached.
> > Running just on only one host is also not working. The output from
> > mpirun using "-d" option for this scenario is attached in file
> > "mpirun_output_d_option_one_host.txt".
>
