Thanks, much appreciated.
On Fri, Aug 31, 2012 at 2:37 PM, Ralph Castain <r...@open-mpi.org> wrote:
> I see - well, I hope to work on it this weekend and may get it fixed. If I
> do, I can provide you with a patch for the 1.6 series that you can use until
> the actual release is issued, if that helps.
>
>
> On Aug 31, 2012, at 2:33 PM, Brian Budge <brian.bu...@gmail.com> wrote:
>
>> Hi Ralph -
>>
>> This is true, but we may not know until well into the process whether
>> we need MPI at all. We have an SMP/NUMA mode that is designed to run
>> faster on a single machine. We also may build our application on
>> machines where there is no MPI, and we simply don't build the code
>> that runs the MPI functionality in that case. We have scripts all
>> over the place that need to start this application, and it would be
>> much easier to be able to simply run the program than to figure out
>> when or if mpirun needs to be starting the program.
>>
>> Before, we went so far as to fork and exec a full mpirun when we run
>> in clustered mode. This resulted in an additional process running,
>> and we had to use sockets to get the data to the new master process.
>> I very much like the idea of being able to have our process become the
>> MPI master instead, so I have been very excited about your work around
>> this singleton fork/exec under the hood.
>>
>> Once I get my new infrastructure designed to work with mpirun -n 1 +
>> spawn, I will try some previous openmpi versions to see if I can find
>> a version with this singleton functionality in-tact.
>>
>> Thanks again,
>> Brian
>>
>> On Thu, Aug 30, 2012 at 4:51 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>> not off the top of my head. However, as noted earlier, there is absolutely
>>> no advantage to a singleton vs mpirun start - all the singleton does is
>>> immediately fork/exec "mpirun" to support the rest of the job. In both
>>> cases, you have a daemon running the job - only difference is in the number
>>> of characters the user types to start it.
>>>
>>>
>>> On Aug 30, 2012, at 8:44 AM, Brian Budge <brian.bu...@gmail.com> wrote:
>>>
>>>> In the event that I need to get this up-and-running soon (I do need
>>>> something working within 2 weeks), can you recommend an older version
>>>> where this is expected to work?
>>>>
>>>> Thanks,
>>>> Brian
>>>>
>>>> On Tue, Aug 28, 2012 at 4:58 PM, Brian Budge <brian.bu...@gmail.com> wrote:
>>>>> Thanks!
>>>>>
>>>>> On Tue, Aug 28, 2012 at 4:57 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>> Yeah, I'm seeing the hang as well when running across multiple machines.
>>>>>> Let me dig a little and get this fixed.
>>>>>>
>>>>>> Thanks
>>>>>> Ralph
>>>>>>
>>>>>> On Aug 28, 2012, at 4:51 PM, Brian Budge <brian.bu...@gmail.com> wrote:
>>>>>>
>>>>>>> Hmmm, I went to the build directories of openmpi for my two machines,
>>>>>>> went into the orte/test/mpi directory and made the executables on both
>>>>>>> machines. I set the hostsfile in the env variable on the "master"
>>>>>>> machine.
>>>>>>>
>>>>>>> Here's the output:
>>>>>>>
>>>>>>> OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile
>>>>>>> ./simple_spawn
>>>>>>> Parent [pid 97504] starting up!
>>>>>>> 0 completed MPI_Init
>>>>>>> Parent [pid 97504] about to spawn!
>>>>>>> Parent [pid 97507] starting up!
>>>>>>> Parent [pid 97508] starting up!
>>>>>>> Parent [pid 30626] starting up!
>>>>>>> ^C
>>>>>>> zsh: interrupt OMPI_MCA_orte_default_hostfile= ./simple_spawn
>>>>>>>
>>>>>>> I had to ^C to kill the hung process.
>>>>>>>
>>>>>>> When I run using mpirun:
>>>>>>>
>>>>>>> OMPI_MCA_orte_default_hostfile=/home/budgeb/p4/pseb/external/install/openmpi-1.6.1/orte/test/mpi/hostsfile
>>>>>>> mpirun -np 1 ./simple_spawn
>>>>>>> Parent [pid 97511] starting up!
>>>>>>> 0 completed MPI_Init
>>>>>>> Parent [pid 97511] about to spawn!
>>>>>>> Parent [pid 97513] starting up!
>>>>>>> Parent [pid 30762] starting up!
>>>>>>> Parent [pid 30764] starting up!
>>>>>>> Parent done with spawn
>>>>>>> Parent sending message to child
>>>>>>> 1 completed MPI_Init
>>>>>>> Hello from the child 1 of 3 on host budgeb-sandybridge pid 97513
>>>>>>> 0 completed MPI_Init
>>>>>>> Hello from the child 0 of 3 on host budgeb-interlagos pid 30762
>>>>>>> 2 completed MPI_Init
>>>>>>> Hello from the child 2 of 3 on host budgeb-interlagos pid 30764
>>>>>>> Child 1 disconnected
>>>>>>> Child 0 received msg: 38
>>>>>>> Child 0 disconnected
>>>>>>> Parent disconnected
>>>>>>> Child 2 disconnected
>>>>>>> 97511: exiting
>>>>>>> 97513: exiting
>>>>>>> 30762: exiting
>>>>>>> 30764: exiting
>>>>>>>
>>>>>>> As you can see, I'm using openmpi v 1.6.1. I just barely freshly
>>>>>>> installed on both machines using the default configure options.
>>>>>>>
>>>>>>> Thanks for all your help.
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>> On Tue, Aug 28, 2012 at 4:39 PM, Ralph Castain <r...@open-mpi.org>
>>>>>>> wrote:
>>>>>>>> Looks to me like it didn't find your executable - could be a question
>>>>>>>> of where it exists relative to where you are running. If you look in
>>>>>>>> your OMPI source tree at the orte/test/mpi directory, you'll see an
>>>>>>>> example program "simple_spawn.c" there. Just "make simple_spawn" and
>>>>>>>> execute that with your default hostfile set - does it work okay?
>>>>>>>>
>>>>>>>> It works fine for me, hence the question.
>>>>>>>>
>>>>>>>> Also, what OMPI version are you using?
>>>>>>>>
>>>>>>>> On Aug 28, 2012, at 4:25 PM, Brian Budge <brian.bu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I see. Okay. So, I just tried removing the check for universe size,
>>>>>>>>> and set the universe size to 2. Here's my output:
>>>>>>>>>
>>>>>>>>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib
>>>>>>>>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile ./master_exe
>>>>>>>>> [budgeb-interlagos:29965] [[4156,0],0] ORTE_ERROR_LOG: Fatal in file
>>>>>>>>> base/plm_base_receive.c at line 253
>>>>>>>>> [budgeb-interlagos:29963] [[4156,1],0] ORTE_ERROR_LOG: The specified
>>>>>>>>> application failed to start in file dpm_orte.c at line 785
>>>>>>>>>
>>>>>>>>> The corresponding run with mpirun still works.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Brian
>>>>>>>>>
>>>>>>>>> On Tue, Aug 28, 2012 at 2:46 PM, Ralph Castain <r...@open-mpi.org>
>>>>>>>>> wrote:
>>>>>>>>>> I see the issue - it's here:
>>>>>>>>>>
>>>>>>>>>>> MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &puniverseSize,
>>>>>>>>>>> &flag);
>>>>>>>>>>>
>>>>>>>>>>> if(!flag) {
>>>>>>>>>>> std::cerr << "no universe size" << std::endl;
>>>>>>>>>>> return -1;
>>>>>>>>>>> }
>>>>>>>>>>> universeSize = *puniverseSize;
>>>>>>>>>>> if(universeSize == 1) {
>>>>>>>>>>> std::cerr << "cannot start slaves... not enough nodes" <<
>>>>>>>>>>> std::endl;
>>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> The universe size is set to 1 on a singleton because the attribute
>>>>>>>>>> gets set at the beginning of time - we haven't any way to go back
>>>>>>>>>> and change it. The sequence of events explains why. The singleton
>>>>>>>>>> starts up and sets its attributes, including universe_size. It also
>>>>>>>>>> spins off an orte daemon to act as its own private "mpirun" in case
>>>>>>>>>> you call comm_spawn. At this point, however, no hostfile has been
>>>>>>>>>> read - the singleton is just an MPI proc doing its own thing, and
>>>>>>>>>> the orte daemon is just sitting there on "stand-by".
>>>>>>>>>>
>>>>>>>>>> When your app calls comm_spawn, then the orte daemon gets called to
>>>>>>>>>> launch the new procs. At that time, it (not the original singleton!)
>>>>>>>>>> reads the hostfile to find out how many nodes are around, and then
>>>>>>>>>> does the launch.
>>>>>>>>>>
>>>>>>>>>> You are trying to check the number of nodes from within the
>>>>>>>>>> singleton, which won't work - it has no way of discovering that info.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Aug 28, 2012, at 2:38 PM, Brian Budge <brian.bu...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> echo hostsfile
>>>>>>>>>>> localhost
>>>>>>>>>>> budgeb-sandybridge
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Brian
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Aug 28, 2012 at 2:36 PM, Ralph Castain <r...@open-mpi.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hmmm...what is in your "hostsfile"?
>>>>>>>>>>>>
>>>>>>>>>>>> On Aug 28, 2012, at 2:33 PM, Brian Budge <brian.bu...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ralph -
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for confirming this is possible. I'm trying this and
>>>>>>>>>>>>> currently
>>>>>>>>>>>>> failing. Perhaps there's something I'm missing in the code to
>>>>>>>>>>>>> make
>>>>>>>>>>>>> this work. Here are the two instantiations and their outputs:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib
>>>>>>>>>>>>>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile ./master_exe
>>>>>>>>>>>>> cannot start slaves... not enough nodes
>>>>>>>>>>>>>
>>>>>>>>>>>>>> LD_LIBRARY_PATH=/home/budgeb/p4/pseb/external/lib.dev:/usr/local/lib
>>>>>>>>>>>>>> OMPI_MCA_orte_default_hostfile=`pwd`/hostsfile mpirun -n 1
>>>>>>>>>>>>>> ./master_exe
>>>>>>>>>>>>> master spawned 1 slaves...
>>>>>>>>>>>>> slave responding...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> The code:
>>>>>>>>>>>>>
>>>>>>>>>>>>> //master.cpp
>>>>>>>>>>>>> #include <mpi.h>
>>>>>>>>>>>>> #include <boost/filesystem.hpp>
>>>>>>>>>>>>> #include <iostream>
>>>>>>>>>>>>>
>>>>>>>>>>>>> int main(int argc, char **args) {
>>>>>>>>>>>>> int worldSize, universeSize, *puniverseSize, flag;
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Comm everyone; //intercomm
>>>>>>>>>>>>> boost::filesystem::path curPath =
>>>>>>>>>>>>> boost::filesystem::absolute(boost::filesystem::current_path());
>>>>>>>>>>>>>
>>>>>>>>>>>>> std::string toRun = (curPath / "slave_exe").string();
>>>>>>>>>>>>>
>>>>>>>>>>>>> int ret = MPI_Init(&argc, &args);
>>>>>>>>>>>>>
>>>>>>>>>>>>> if(ret != MPI_SUCCESS) {
>>>>>>>>>>>>> std::cerr << "failed init" << std::endl;
>>>>>>>>>>>>> return -1;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Comm_size(MPI_COMM_WORLD, &worldSize);
>>>>>>>>>>>>>
>>>>>>>>>>>>> if(worldSize != 1) {
>>>>>>>>>>>>> std::cerr << "too many masters" << std::endl;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Attr_get(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, &puniverseSize,
>>>>>>>>>>>>> &flag);
>>>>>>>>>>>>>
>>>>>>>>>>>>> if(!flag) {
>>>>>>>>>>>>> std::cerr << "no universe size" << std::endl;
>>>>>>>>>>>>> return -1;
>>>>>>>>>>>>> }
>>>>>>>>>>>>> universeSize = *puniverseSize;
>>>>>>>>>>>>> if(universeSize == 1) {
>>>>>>>>>>>>> std::cerr << "cannot start slaves... not enough nodes" <<
>>>>>>>>>>>>> std::endl;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> char *buf = (char*)alloca(toRun.size() + 1);
>>>>>>>>>>>>> memcpy(buf, toRun.c_str(), toRun.size());
>>>>>>>>>>>>> buf[toRun.size()] = '\0';
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Comm_spawn(buf, MPI_ARGV_NULL, universeSize-1, MPI_INFO_NULL,
>>>>>>>>>>>>> 0, MPI_COMM_SELF, &everyone,
>>>>>>>>>>>>> MPI_ERRCODES_IGNORE);
>>>>>>>>>>>>>
>>>>>>>>>>>>> std::cerr << "master spawned " << universeSize-1 << " slaves..."
>>>>>>>>>>>>> << std::endl;
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Finalize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> return 0;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> //slave.cpp
>>>>>>>>>>>>> #include <mpi.h>
>>>>>>>>>>>>>
>>>>>>>>>>>>> int main(int argc, char **args) {
>>>>>>>>>>>>> int size;
>>>>>>>>>>>>> MPI_Comm parent;
>>>>>>>>>>>>> MPI_Init(&argc, &args);
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Comm_get_parent(&parent);
>>>>>>>>>>>>>
>>>>>>>>>>>>> if(parent == MPI_COMM_NULL) {
>>>>>>>>>>>>> std::cerr << "slave has no parent" << std::endl;
>>>>>>>>>>>>> }
>>>>>>>>>>>>> MPI_Comm_remote_size(parent, &size);
>>>>>>>>>>>>> if(size != 1) {
>>>>>>>>>>>>> std::cerr << "parent size is " << size << std::endl;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> std::cerr << "slave responding..." << std::endl;
>>>>>>>>>>>>>
>>>>>>>>>>>>> MPI_Finalize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> return 0;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any ideas? Thanks for any help.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Brian
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 22, 2012 at 9:03 AM, Ralph Castain
>>>>>>>>>>>>> <r...@open-mpi.org> wrote:
>>>>>>>>>>>>>> It really is just that simple :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Aug 22, 2012, at 8:56 AM, Brian Budge <brian.bu...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Okay. Is there a tutorial or FAQ for setting everything up?
>>>>>>>>>>>>>>> Or is it
>>>>>>>>>>>>>>> really just that simple? I don't need to run a copy of the orte
>>>>>>>>>>>>>>> server somewhere?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if my current ip is 192.168.0.1,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 0 > echo 192.168.0.11 > /tmp/hostfile
>>>>>>>>>>>>>>> 1 > echo 192.168.0.12 >> /tmp/hostfile
>>>>>>>>>>>>>>> 2 > export OMPI_MCA_orte_default_hostfile=/tmp/hostfile
>>>>>>>>>>>>>>> 3 > ./mySpawningExe
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At this point, mySpawningExe will be the master, running on
>>>>>>>>>>>>>>> 192.168.0.1, and I can have spawned, for example, childExe on
>>>>>>>>>>>>>>> 192.168.0.11 and 192.168.0.12? Or childExe1 on 192.168.0.11 and
>>>>>>>>>>>>>>> childExe2 on 192.168.0.12?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the help.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Brian
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 22, 2012 at 7:15 AM, Ralph Castain
>>>>>>>>>>>>>>> <r...@open-mpi.org> wrote:
>>>>>>>>>>>>>>>> Sure, that's still true on all 1.3 or above releases. All you
>>>>>>>>>>>>>>>> need to do is set the hostfile envar so we pick it up:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> OMPI_MCA_orte_default_hostfile=<foo>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Aug 21, 2012, at 7:23 PM, Brian Budge
>>>>>>>>>>>>>>>> <brian.bu...@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi. I know this is an old thread, but I'm curious if there
>>>>>>>>>>>>>>>>> are any
>>>>>>>>>>>>>>>>> tutorials describing how to set this up? Is this still
>>>>>>>>>>>>>>>>> available on
>>>>>>>>>>>>>>>>> newer open mpi versions?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Brian
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jan 4, 2008 at 7:57 AM, Ralph Castain <r...@lanl.gov>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Hi Elena
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm copying this to the user list just to correct a
>>>>>>>>>>>>>>>>>> mis-statement on my part
>>>>>>>>>>>>>>>>>> in an earlier message that went there. I had stated that a
>>>>>>>>>>>>>>>>>> singleton could
>>>>>>>>>>>>>>>>>> comm_spawn onto other nodes listed in a hostfile by setting
>>>>>>>>>>>>>>>>>> an environmental
>>>>>>>>>>>>>>>>>> variable that pointed us to the hostfile.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is incorrect in the 1.2 code series. That series does
>>>>>>>>>>>>>>>>>> not allow
>>>>>>>>>>>>>>>>>> singletons to read a hostfile at all. Hence, any comm_spawn
>>>>>>>>>>>>>>>>>> done by a
>>>>>>>>>>>>>>>>>> singleton can only launch child processes on the singleton's
>>>>>>>>>>>>>>>>>> local host.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This situation has been corrected for the upcoming 1.3 code
>>>>>>>>>>>>>>>>>> series. For the
>>>>>>>>>>>>>>>>>> 1.2 series, though, you will have to do it via an mpirun
>>>>>>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sorry for the confusion - I sometimes have too many code
>>>>>>>>>>>>>>>>>> families to keep
>>>>>>>>>>>>>>>>>> straight in this old mind!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 1/4/08 5:10 AM, "Elena Zhebel" <ezhe...@fugro-jason.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello Ralph,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you very much for the explanations.
>>>>>>>>>>>>>>>>>>> But I still do not get it running...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For the case
>>>>>>>>>>>>>>>>>>> mpirun -n 1 -hostfile my_hostfile -host my_master_host
>>>>>>>>>>>>>>>>>>> my_master.exe
>>>>>>>>>>>>>>>>>>> everything works.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For the case
>>>>>>>>>>>>>>>>>>> ./my_master.exe
>>>>>>>>>>>>>>>>>>> it does not.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I did:
>>>>>>>>>>>>>>>>>>> - create my_hostfile and put it in the
>>>>>>>>>>>>>>>>>>> $HOME/.openmpi/components/
>>>>>>>>>>>>>>>>>>> my_hostfile :
>>>>>>>>>>>>>>>>>>> bollenstreek slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> octocore01 slots=8 max_slots=8
>>>>>>>>>>>>>>>>>>> octocore02 slots=8 max_slots=8
>>>>>>>>>>>>>>>>>>> clstr000 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr001 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr002 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr003 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr004 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr005 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr006 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> clstr007 slots=2 max_slots=3
>>>>>>>>>>>>>>>>>>> - setenv OMPI_MCA_rds_hostfile_path my_hostfile (I put it
>>>>>>>>>>>>>>>>>>> in .tcshrc and
>>>>>>>>>>>>>>>>>>> then source .tcshrc)
>>>>>>>>>>>>>>>>>>> - in my_master.cpp I did
>>>>>>>>>>>>>>>>>>> MPI_Info info1;
>>>>>>>>>>>>>>>>>>> MPI_Info_create(&info1);
>>>>>>>>>>>>>>>>>>> char* hostname =
>>>>>>>>>>>>>>>>>>> "clstr002,clstr003,clstr005,clstr006,clstr007,octocore01,octocore02";
>>>>>>>>>>>>>>>>>>> MPI_Info_set(info1, "host", hostname);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> _intercomm = intracomm.Spawn("./childexe", argv1, _nProc,
>>>>>>>>>>>>>>>>>>> info1, 0,
>>>>>>>>>>>>>>>>>>> MPI_ERRCODES_IGNORE);
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - After I call the executable, I've got this error message
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> bollenstreek: > ./my_master
>>>>>>>>>>>>>>>>>>> number of processes to run: 1
>>>>>>>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> Some of the requested hosts are not included in the current
>>>>>>>>>>>>>>>>>>> allocation for
>>>>>>>>>>>>>>>>>>> the application:
>>>>>>>>>>>>>>>>>>> ./childexe
>>>>>>>>>>>>>>>>>>> The requested hosts were:
>>>>>>>>>>>>>>>>>>> clstr002,clstr003,clstr005,clstr006,clstr007,octocore01,octocore02
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Verify that you have mapped the allocated resources
>>>>>>>>>>>>>>>>>>> properly using the
>>>>>>>>>>>>>>>>>>> --host specification.
>>>>>>>>>>>>>>>>>>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of
>>>>>>>>>>>>>>>>>>> resource in file
>>>>>>>>>>>>>>>>>>> base/rmaps_base_support_fns.c at line 225
>>>>>>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of
>>>>>>>>>>>>>>>>>>> resource in file
>>>>>>>>>>>>>>>>>>> rmaps_rr.c at line 478
>>>>>>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of
>>>>>>>>>>>>>>>>>>> resource in file
>>>>>>>>>>>>>>>>>>> base/rmaps_base_map_job.c at line 210
>>>>>>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of
>>>>>>>>>>>>>>>>>>> resource in file
>>>>>>>>>>>>>>>>>>> rmgr_urm.c at line 372
>>>>>>>>>>>>>>>>>>> [bollenstreek:21443] [0,0,0] ORTE_ERROR_LOG: Out of
>>>>>>>>>>>>>>>>>>> resource in file
>>>>>>>>>>>>>>>>>>> communicator/comm_dyn.c at line 608
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Did I miss something?
>>>>>>>>>>>>>>>>>>> Thanks for help!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Elena
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>> From: Ralph H Castain [mailto:r...@lanl.gov]
>>>>>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2007 3:50 PM
>>>>>>>>>>>>>>>>>>> To: Elena Zhebel; Open MPI Users <us...@open-mpi.org>
>>>>>>>>>>>>>>>>>>> Cc: Ralph H Castain
>>>>>>>>>>>>>>>>>>> Subject: Re: [OMPI users] MPI::Intracomm::Spawn and cluster
>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 12/18/07 7:35 AM, "Elena Zhebel"
>>>>>>>>>>>>>>>>>>> <ezhe...@fugro-jason.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks a lot! Now it works!
>>>>>>>>>>>>>>>>>>>> The solution is to use mpirun -n 1 -hostfile my.hosts
>>>>>>>>>>>>>>>>>>>> *.exe and pass
>>>>>>>>>>>>>>>>>>> MPI_Info
>>>>>>>>>>>>>>>>>>>> Key to the Spawn function!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> One more question: is it necessary to start my "master"
>>>>>>>>>>>>>>>>>>>> program with
>>>>>>>>>>>>>>>>>>>> mpirun -n 1 -hostfile my_hostfile -host my_master_host
>>>>>>>>>>>>>>>>>>>> my_master.exe ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> No, it isn't necessary - assuming that my_master_host is
>>>>>>>>>>>>>>>>>>> the first host
>>>>>>>>>>>>>>>>>>> listed in your hostfile! If you are only executing one
>>>>>>>>>>>>>>>>>>> my_master.exe (i.e.,
>>>>>>>>>>>>>>>>>>> you gave -n 1 to mpirun), then we will automatically map
>>>>>>>>>>>>>>>>>>> that process onto
>>>>>>>>>>>>>>>>>>> the first host in your hostfile.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If you want my_master.exe to go on someone other than the
>>>>>>>>>>>>>>>>>>> first host in the
>>>>>>>>>>>>>>>>>>> file, then you have to give us the -host option.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Are there other possibilities for easy start?
>>>>>>>>>>>>>>>>>>>> I would say just to run ./my_master.exe , but then the
>>>>>>>>>>>>>>>>>>>> master process
>>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>> know about the available in the network hosts.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You can set the hostfile parameter in your environment
>>>>>>>>>>>>>>>>>>> instead of on the
>>>>>>>>>>>>>>>>>>> command line. Just set OMPI_MCA_rds_hostfile_path =
>>>>>>>>>>>>>>>>>>> my.hosts.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> You can then just run ./my_master.exe on the host where you
>>>>>>>>>>>>>>>>>>> want the master
>>>>>>>>>>>>>>>>>>> to reside - everything should work the same.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just as an FYI: the name of that environmental variable is
>>>>>>>>>>>>>>>>>>> going to change
>>>>>>>>>>>>>>>>>>> in the 1.3 release, but everything will still work the same.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hope that helps
>>>>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks and regards,
>>>>>>>>>>>>>>>>>>>> Elena
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>> From: Ralph H Castain [mailto:r...@lanl.gov]
>>>>>>>>>>>>>>>>>>>> Sent: Monday, December 17, 2007 5:49 PM
>>>>>>>>>>>>>>>>>>>> To: Open MPI Users <us...@open-mpi.org>; Elena Zhebel
>>>>>>>>>>>>>>>>>>>> Cc: Ralph H Castain
>>>>>>>>>>>>>>>>>>>> Subject: Re: [OMPI users] MPI::Intracomm::Spawn and
>>>>>>>>>>>>>>>>>>>> cluster configuration
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 12/17/07 8:19 AM, "Elena Zhebel"
>>>>>>>>>>>>>>>>>>>> <ezhe...@fugro-jason.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hello Ralph,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you for your answer.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm using OpenMPI 1.2.3. , compiler glibc232, Linux Suse
>>>>>>>>>>>>>>>>>>>>> 10.0.
>>>>>>>>>>>>>>>>>>>>> My "master" executable runs only on the one local host,
>>>>>>>>>>>>>>>>>>>>> then it spawns
>>>>>>>>>>>>>>>>>>>>> "slaves" (with MPI::Intracomm::Spawn).
>>>>>>>>>>>>>>>>>>>>> My question was: how to determine the hosts where these
>>>>>>>>>>>>>>>>>>>>> "slaves" will be
>>>>>>>>>>>>>>>>>>>>> spawned?
>>>>>>>>>>>>>>>>>>>>> You said: "You have to specify all of the hosts that can
>>>>>>>>>>>>>>>>>>>>> be used by
>>>>>>>>>>>>>>>>>>>>> your job
>>>>>>>>>>>>>>>>>>>>> in the original hostfile". How can I specify the host
>>>>>>>>>>>>>>>>>>>>> file? I can not
>>>>>>>>>>>>>>>>>>>>> find it
>>>>>>>>>>>>>>>>>>>>> in the documentation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hmmm...sorry about the lack of documentation. I always
>>>>>>>>>>>>>>>>>>>> assumed that the MPI
>>>>>>>>>>>>>>>>>>>> folks in the project would document such things since it
>>>>>>>>>>>>>>>>>>>> has little to do
>>>>>>>>>>>>>>>>>>>> with the underlying run-time, but I guess that fell
>>>>>>>>>>>>>>>>>>>> through the cracks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> There are two parts to your question:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1. how to specify the hosts to be used for the entire job.
>>>>>>>>>>>>>>>>>>>> I believe that
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> somewhat covered here:
>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/faq/?category=running#simple-spmd-run
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> That FAQ tells you what a hostfile should look like,
>>>>>>>>>>>>>>>>>>>> though you may already
>>>>>>>>>>>>>>>>>>>> know that. Basically, we require that you list -all- of
>>>>>>>>>>>>>>>>>>>> the nodes that both
>>>>>>>>>>>>>>>>>>>> your master and slave programs will use.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2. how to specify which nodes are available for the
>>>>>>>>>>>>>>>>>>>> master, and which for
>>>>>>>>>>>>>>>>>>>> the slave.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You would specify the host for your master on the mpirun
>>>>>>>>>>>>>>>>>>>> command line with
>>>>>>>>>>>>>>>>>>>> something like:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> mpirun -n 1 -hostfile my_hostfile -host my_master_host
>>>>>>>>>>>>>>>>>>>> my_master.exe
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This directs Open MPI to map that specified executable on
>>>>>>>>>>>>>>>>>>>> the specified
>>>>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>>>>>> - note that my_master_host must have been in my_hostfile.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Inside your master, you would create an MPI_Info key
>>>>>>>>>>>>>>>>>>>> "host" that has a
>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>> consisting of a string "host1,host2,host3" identifying the
>>>>>>>>>>>>>>>>>>>> hosts you want
>>>>>>>>>>>>>>>>>>>> your slave to execute upon. Those hosts must have been
>>>>>>>>>>>>>>>>>>>> included in
>>>>>>>>>>>>>>>>>>>> my_hostfile. Include that key in the MPI_Info array passed
>>>>>>>>>>>>>>>>>>>> to your Spawn.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We don't currently support providing a hostfile for the
>>>>>>>>>>>>>>>>>>>> slaves (as opposed
>>>>>>>>>>>>>>>>>>>> to the host-at-a-time string above). This may become
>>>>>>>>>>>>>>>>>>>> available in a future
>>>>>>>>>>>>>>>>>>>> release - TBD.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hope that helps
>>>>>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks and regards,
>>>>>>>>>>>>>>>>>>>>> Elena
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>>> From: users-boun...@open-mpi.org
>>>>>>>>>>>>>>>>>>>>> [mailto:users-boun...@open-mpi.org] On
>>>>>>>>>>>>>>>>>>>>> Behalf Of Ralph H Castain
>>>>>>>>>>>>>>>>>>>>> Sent: Monday, December 17, 2007 3:31 PM
>>>>>>>>>>>>>>>>>>>>> To: Open MPI Users <us...@open-mpi.org>
>>>>>>>>>>>>>>>>>>>>> Cc: Ralph H Castain
>>>>>>>>>>>>>>>>>>>>> Subject: Re: [OMPI users] MPI::Intracomm::Spawn and
>>>>>>>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 12/12/07 5:46 AM, "Elena Zhebel"
>>>>>>>>>>>>>>>>>>>>> <ezhe...@fugro-jason.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I'm working on a MPI application where I'm using OpenMPI
>>>>>>>>>>>>>>>>>>>>>> instead of
>>>>>>>>>>>>>>>>>>>>>> MPICH.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In my "master" program I call the function
>>>>>>>>>>>>>>>>>>>>>> MPI::Intracomm::Spawn which
>>>>>>>>>>>>>>>>>>>>> spawns
>>>>>>>>>>>>>>>>>>>>>> "slave" processes. It is not clear for me how to spawn
>>>>>>>>>>>>>>>>>>>>>> the "slave"
>>>>>>>>>>>>>>>>>>>>> processes
>>>>>>>>>>>>>>>>>>>>>> over the network. Currently "master" creates "slaves" on
>>>>>>>>>>>>>>>>>>>>>> the same
>>>>>>>>>>>>>>>>>>>>>> host.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If I use 'mpirun --hostfile openmpi.hosts' then
>>>>>>>>>>>>>>>>>>>>>> processes are spawn
>>>>>>>>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> network as expected. But now I need to spawn processes
>>>>>>>>>>>>>>>>>>>>>> over the
>>>>>>>>>>>>>>>>>>>>>> network
>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>> my own executable using MPI::Intracomm::Spawn, how can I
>>>>>>>>>>>>>>>>>>>>>> achieve it?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm not sure from your description exactly what you are
>>>>>>>>>>>>>>>>>>>>> trying to do,
>>>>>>>>>>>>>>>>>>>>> nor in
>>>>>>>>>>>>>>>>>>>>> what environment this is all operating within or what
>>>>>>>>>>>>>>>>>>>>> version of Open
>>>>>>>>>>>>>>>>>>>>> MPI
>>>>>>>>>>>>>>>>>>>>> you are using. Setting aside the environment and version
>>>>>>>>>>>>>>>>>>>>> issue, I'm
>>>>>>>>>>>>>>>>>>>>> guessing
>>>>>>>>>>>>>>>>>>>>> that you are running your executable over some specified
>>>>>>>>>>>>>>>>>>>>> set of hosts,
>>>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> want to provide a different hostfile that specifies the
>>>>>>>>>>>>>>>>>>>>> hosts to be
>>>>>>>>>>>>>>>>>>>>> used for
>>>>>>>>>>>>>>>>>>>>> the "slave" processes. Correct?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> If that is correct, then I'm afraid you can't do that in
>>>>>>>>>>>>>>>>>>>>> any version
>>>>>>>>>>>>>>>>>>>>> of Open
>>>>>>>>>>>>>>>>>>>>> MPI today. You have to specify all of the hosts that can
>>>>>>>>>>>>>>>>>>>>> be used by
>>>>>>>>>>>>>>>>>>>>> your job
>>>>>>>>>>>>>>>>>>>>> in the original hostfile. You can then specify a subset
>>>>>>>>>>>>>>>>>>>>> of those hosts
>>>>>>>>>>>>>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>>>> used by your original "master" program, and then specify
>>>>>>>>>>>>>>>>>>>>> a different
>>>>>>>>>>>>>>>>>>>>> subset
>>>>>>>>>>>>>>>>>>>>> to be used by the "slaves" when calling Spawn.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> But the system requires that you tell it -all- of the
>>>>>>>>>>>>>>>>>>>>> hosts that are
>>>>>>>>>>>>>>>>>>>>> going
>>>>>>>>>>>>>>>>>>>>> to be used at the beginning of the job.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> At the moment, there is no plan to remove that
>>>>>>>>>>>>>>>>>>>>> requirement, though
>>>>>>>>>>>>>>>>>>>>> there has
>>>>>>>>>>>>>>>>>>>>> been occasional discussion about doing so at some point
>>>>>>>>>>>>>>>>>>>>> in the future.
>>>>>>>>>>>>>>>>>>>>> No
>>>>>>>>>>>>>>>>>>>>> promises that it will happen, though - managed
>>>>>>>>>>>>>>>>>>>>> environments, in
>>>>>>>>>>>>>>>>>>>>> particular,
>>>>>>>>>>>>>>>>>>>>> currently object to the idea of changing the allocation
>>>>>>>>>>>>>>>>>>>>> on-the-fly. We
>>>>>>>>>>>>>>>>>>>>> may,
>>>>>>>>>>>>>>>>>>>>> though, make a provision for purely hostfile-based
>>>>>>>>>>>>>>>>>>>>> environments (i.e.,
>>>>>>>>>>>>>>>>>>>>> unmanaged) at some time in the future.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks in advance for any help.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Elena
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> users mailing list
>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> us...@open-mpi.org
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
