Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-15 Thread Jeff Squyres
the same. Again, thanks a lot guys for your invaluable help. Regards, Prasanna. On 9/15/08 11:08 AM, "users-requ...@open-mpi.org" wrote: Message: 1 Date: Mon, 15 Sep 2008 12:42:50 -0400 From: Eric Thibodeau Subject: Re: [OMPI users] Need help resolving No route to host error with Ope

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-15 Thread Prasanna Ranganathan
i.org" wrote: > Message: 1 > Date: Mon, 15 Sep 2008 12:42:50 -0400 > From: Eric Thibodeau > Subject: Re: [OMPI users] Need help resolving No route to host error > with OpenMPI 1.1.2 > To: Open MPI Users > Message-ID: <48ce908a.9080...@neuralbs.com> > Content-Type:

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-15 Thread Eric Thibodeau
Simply to keep track of what's going on: I checked the build environment for openmpi and the system's setting, they were built using gcc 3.4.4 with -Os, which was reputed unstable and problematic with this compiler version. I've asked Prasanna to rebuild using -O2 but this could be a bit lengt

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-12 Thread Eric Thibodeau
Prasanna, Please send me your /etc/make.conf and the contents of /var/db/pkg/sys-cluster/openmpi-1.2.7/ You can package this with the following command line: tar -cjf data.tbz /etc/make.conf /var/db/pkg/sys-cluster/openmpi-1.2.7/ And simply send me the data.tbz file. Thanks, Eric Prasa

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-12 Thread Prasanna Ranganathan
Hi, I did make sure at the beginning that only eth0 was activated on all the nodes. Nevertheless, I am currently verifying the NIC configuration on all the nodes and making sure things are as expected. While trying different things, I did come across this peculiar error which I had detailed in o

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-12 Thread Matt Hughes
Hi Prasanna, do you have any unusual ethernet interfaces on your nodes? I have seen similar problems when using IP over Infiniband. I'm not sure exactly why, but mixing interfaces of different types (ib0 and eth0 for example) can sometimes cause these problems, possibly because they are on differe

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-12 Thread Prasanna Ranganathan
Hi, I have verified the openMPI version to be 1.2.7 on all the nodes and also ompi_info | grep thread is Thread support: posix (mpi: no, progress: no) on these machines. I get the error with and without -mca oob_tcp_listen_mode listen_thread. Sometimes, the startup takes too long with the liste

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-12 Thread Jeff Squyres
On Sep 11, 2008, at 6:29 PM, Prasanna Ranganathan wrote: I have tried the following to no avail. On 499 machines running openMPI 1.2.7: mpirun -np 499 -bynode -hostfile nodelist /main/mpiHelloWorld ... With different combinations of the following parameters -mca btl_base_verbose 1 -mca btl_b

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Eric Thibodeau
is to do with the openMPI 1.2.7 having posix enabled in the current configuration on these nodes? Thanks again for your continued help. Regards, Prasanna. Message: 2 Date: Thu, 11 Sep 2008 12:16:50 -0400 From: Jeff Squyres Subject: Re: [OMPI users] Need help resolving No route to ho

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Eric Thibodeau
Jeff Squyres wrote: On Sep 11, 2008, at 3:27 PM, Eric Thibodeau wrote: Ok, added to the information from the README, I'm thinking none of the 3 configure options have an impact on the said 'threaded TCP listener' and the MCA option you suggested should still work, is this correct? It should

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Prasanna Ranganathan
again for your continued help. Regards, Prasanna. > Message: 2 > Date: Thu, 11 Sep 2008 12:16:50 -0400 > From: Jeff Squyres > Subject: Re: [OMPI users] Need help resolving No route to host error > with OpenMPI 1.1.2 > To: Open MPI Users > Message-ID: <7110e2d0-

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Jeff Squyres
On Sep 11, 2008, at 3:27 PM, Eric Thibodeau wrote: Ok, added to the information from the README, I'm thinking none of the 3 configure options have an impact on the said 'threaded TCP listener' and the MCA option you suggested should still work, is this correct? It should default to --with

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Eric Thibodeau
Jeff Squyres wrote: On Sep 11, 2008, at 2:38 PM, Eric Thibodeau wrote: In short: Which of the 3 options is the one known to be unstable in the following: --enable-mpi-threadsEnable threads for MPI applications (default: disabled) --enable-progress-threads

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Jeff Squyres
On Sep 11, 2008, at 2:38 PM, Eric Thibodeau wrote: In short: Which of the 3 options is the one known to be unstable in the following: --enable-mpi-threadsEnable threads for MPI applications (default: disabled) --enable-progress-threads Ena

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Ralph Castain
The two configuration options that are disabled by default (--enable- mpi-threads and --enable-progress-threads) are both known unstable The runtime listen_thread option is quite different and is known safe. Ralph On Sep 11, 2008, at 12:38 PM, Eric Thibodeau wrote: Jeff, In short: Which o

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Eric Thibodeau
Jeff, In short: Which of the 3 options is the one known to be unstable in the following: --enable-mpi-threadsEnable threads for MPI applications (default: disabled) --enable-progress-threads Enable threads asynchronous communication progre

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Jeff Squyres
On Sep 10, 2008, at 9:29 PM, Prasanna Ranganathan wrote: I have upgraded to 1.2.7 and am still noticing the issue. FWIW, we didn't change anything with regards to OOB and TCP from 1.2.6 -> 1.2.7, but it's still good to be at the latest version. Try running with this MCA parameter: mpi

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Eric Thibodeau
s-requ...@open-mpi.org" wrote: Message: 1 Date: Mon, 8 Sep 2008 16:43:33 -0400 From: Jeff Squyres Subject: Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2 To: Open MPI Users Message-ID: Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=y

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-11 Thread Jeff Squyres
f the test program and occurs only certain times. How do I take care of this? Regards, Prasanna. On 9/9/08 9:00 AM, "users-requ...@open-mpi.org" > wrote: Message: 1 Date: Mon, 8 Sep 2008 16:43:33 -0400 From: Jeff Squyres Subject: Re: [OMPI users] Need help resolving No r

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-10 Thread Eric Thibodeau
Prasanna Ranganathan wrote: Hi Eric, Thanks a lot for the reply. I am currently working on upgrading to 1.2.7 I do not quite follow your directions; What do you refer to when you say say "try with USE=-threads..." I am referring to the USE variable which is used to set global package speci

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-10 Thread Prasanna Ranganathan
Hi, I have upgraded to 1.2.7 and am still noticing the issue. Kindly help. > > Message: 1 > Date: Mon, 8 Sep 2008 16:43:33 -0400 > From: Jeff Squyres > Subject: Re: [OMPI users] Need help resolving No route to host error > withOpenMPI 1.1.2 > To: Open MPI Users >

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-10 Thread Prasanna Ranganathan
Hi Eric, Thanks a lot for the reply. I am currently working on upgrading to 1.2.7 I do not quite follow your directions; What do you refer to when you say say "try with USE=-threads..." Kindly excuse if it is a silly question and pardon my ignorance :D Regards, Prasanna.

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-10 Thread Eric Thibodeau
es Subject: Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2 To: Open MPI Users Message-ID: Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes Are you able to upgrade to Open MPI v1.2.7? There were *many* bug fixes and changes in the 1.2 series

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-10 Thread Eric Thibodeau
care of this? Regards, Prasanna. On 9/9/08 9:00 AM, "users-requ...@open-mpi.org" wrote: Message: 1 Date: Mon, 8 Sep 2008 16:43:33 -0400 From: Jeff Squyres Subject: Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2 To: Open MPI Users Message-ID: Co

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-10 Thread Prasanna Ranganathan
ot; wrote: > > Message: 1 > Date: Mon, 8 Sep 2008 16:43:33 -0400 > From: Jeff Squyres > Subject: Re: [OMPI users] Need help resolving No route to host error > with OpenMPI 1.1.2 > To: Open MPI Users > Message-ID: > Content-Type: text/plain; charset=WINDOWS-1252; format=flo

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-09 Thread Prasanna Ranganathan
Hi Jeff/Paul, Thanks a lot for your replies. I am looking into upgrading MPI to a newer version. As I use a few custom built libraries as part of my main parallel application that recommend the use of 1.1.2, I first need to check compatibility issues with the newer version before I can upgrade.

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-09 Thread Paul Kapinos
Hi, First, consider to update to newer OpenMPI. Second, look on your environment on the box you startts OpenMPI (runs mpirun ...). Type ulimit -n to explore how many file descriptors your envirinment have. (ulimit -a for all limits). Note, every process on older versions of OpenMPI (prior 1

Re: [OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-08 Thread Jeff Squyres
Are you able to upgrade to Open MPI v1.2.7? There were *many* bug fixes and changes in the 1.2 series compared to the 1.1 series, some, in particular, were dealing with TCP socket timeouts (which are important when dealing with large numbers of MPI processes). On Sep 8, 2008, at 4:36 PM

[OMPI users] Need help resolving No route to host error with OpenMPI 1.1.2

2008-09-08 Thread Prasanna Ranganathan
Hi, I am trying to run a test mpiHelloWorld program that simply initializes the MPI environment on all the nodes, prints the hostname and rank of each node in the MPI process group and exits. I am using MPI 1.1.2 and am running 997 processes on 499 nodes (Nodes have 2 dual core CPUs). I get the