Seems like connected problem:
I can't use rankfile with app, even after all those fixes ( working with
trunk 1.4a1r21657).
This is my case :
$cat rankfile
rank 0=+n1 slot=0
rank 1=+n0 slot=0
$cat appfile
-np 1 hostname
-np 1 hostname
$mpirun -np 2 -H witch1,witch2 -rf rankfile -app appfile
--------------------------------------------------------------------------
Rankfile claimed host +n1 by index that is bigger than number of allocated
hosts.
--------------------------------------------------------------------------
[dellix7:13414] [[10851,0],0] ORTE_ERROR_LOG: Bad parameter in file
../../../../../orte/mca/rmaps/rank_file/rmaps_rank_file.c at line 422
[dellix7:13414] [[10851,0],0] ORTE_ERROR_LOG: Bad parameter in file
../../../../orte/mca/rmaps/base/rmaps_base_map_job.c at line 85
[dellix7:13414] [[10851,0],0] ORTE_ERROR_LOG: Bad parameter in file
../../../../orte/mca/plm/base/plm_base_launch_support.c at line 103
[dellix7:13414] [[10851,0],0] ORTE_ERROR_LOG: Bad parameter in file
../../../../../orte/mca/plm/rsh/plm_rsh_module.c at line 1001
The problem is, that rankfile mapper tries to find an appropriate host in
the partial ( and not full ) hostlist.
Any suggestions how to fix it?
Thanks
Lenny.
On Wed, May 13, 2009 at 1:55 AM, Ralph Castain <r...@open-mpi.org> wrote:
> Okay, I fixed this today too....r21219
>
>
>
> On May 11, 2009, at 11:27 PM, Anton Starikov wrote:
>
> Now there is another problem :)
>>
>> You can try oversubscribe node. At least by 1 task.
>> If you hostfile and rank file limit you at N procs, you can ask mpirun for
>> N+1 and it wil be not rejected.
>> Although in reality there will be N tasks.
>> So, if your hostfile limit is 4, then "mpirun -np 4" and "mpirun -np 5"
>> both works, but in both cases there are only 4 tasks. It isn't crucial,
>> because there is nor real oversubscription, but there is still some bug
>> which can affect something in future.
>>
>> --
>> Anton Starikov.
>>
>> On May 12, 2009, at 1:45 AM, Ralph Castain wrote:
>>
>> This is fixed as of r21208.
>>>
>>> Thanks for reporting it!
>>> Ralph
>>>
>>>
>>> On May 11, 2009, at 12:51 PM, Anton Starikov wrote:
>>>
>>> Although removing this check solves problem of having more slots in
>>>> rankfile than necessary, there is another problem.
>>>>
>>>> If I set rmaps_base_no_oversubscribe=1 then if, for example:
>>>>
>>>>
>>>> hostfile:
>>>>
>>>> node01
>>>> node01
>>>> node02
>>>> node02
>>>>
>>>> rankfile:
>>>>
>>>> rank 0=node01 slot=1
>>>> rank 1=node01 slot=0
>>>> rank 2=node02 slot=1
>>>> rank 3=node02 slot=0
>>>>
>>>> mpirun -np 4 ./something
>>>>
>>>> complains with:
>>>>
>>>> "There are not enough slots available in the system to satisfy the 4
>>>> slots
>>>> that were requested by the application"
>>>>
>>>> but "mpirun -np 3 ./something" will work though. It works, when you ask
>>>> for 1 CPU less. And the same behavior in any case (shared nodes, non-shared
>>>> nodes, multi-node)
>>>>
>>>> If you switch off rmaps_base_no_oversubscribe, then it works and all
>>>> affinities set as it requested in rankfile, there is no oversubscription.
>>>>
>>>>
>>>> Anton.
>>>>
>>>> On May 5, 2009, at 3:08 PM, Ralph Castain wrote:
>>>>
>>>> Ah - thx for catching that, I'll remove that check. It no longer is
>>>>> required.
>>>>>
>>>>> Thx!
>>>>>
>>>>> On Tue, May 5, 2009 at 7:04 AM, Lenny Verkhovsky <
>>>>> lenny.verkhov...@gmail.com> wrote:
>>>>> According to the code it does cares.
>>>>>
>>>>> $vi orte/mca/rmaps/rank_file/rmaps_rank_file.c +572
>>>>>
>>>>> ival = orte_rmaps_rank_file_value.ival;
>>>>> if ( ival > (np-1) ) {
>>>>> orte_show_help("help-rmaps_rank_file.txt", "bad-rankfile", true, ival,
>>>>> rankfile);
>>>>> rc = ORTE_ERR_BAD_PARAM;
>>>>> goto unlock;
>>>>> }
>>>>>
>>>>> If I remember correctly, I used an array to map ranks, and since the
>>>>> length of array is NP, maximum index must be less than np, so if you have
>>>>> the number of rank > NP, you have no place to put it inside array.
>>>>>
>>>>> "Likewise, if you have more procs than the rankfile specifies, we map
>>>>> the additional procs either byslot (default) or bynode (if you specify
>>>>> that
>>>>> option). So the rankfile doesn't need to contain an entry for every proc."
>>>>> - Correct point.
>>>>>
>>>>>
>>>>> Lenny.
>>>>>
>>>>>
>>>>> On 5/5/09, Ralph Castain <r...@open-mpi.org> wrote: Sorry Lenny, but
>>>>> that isn't correct. The rankfile mapper doesn't care if the rankfile
>>>>> contains additional info - it only maps up to the number of processes, and
>>>>> ignores anything beyond that number. So there is no need to remove the
>>>>> additional info.
>>>>>
>>>>> Likewise, if you have more procs than the rankfile specifies, we map
>>>>> the additional procs either byslot (default) or bynode (if you specify
>>>>> that
>>>>> option). So the rankfile doesn't need to contain an entry for every proc.
>>>>>
>>>>> Just don't want to confuse folks.
>>>>> Ralph
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 5, 2009 at 5:59 AM, Lenny Verkhovsky <
>>>>> lenny.verkhov...@gmail.com> wrote:
>>>>> Hi,
>>>>> maximum rank number must be less then np.
>>>>> if np=1 then there is only rank 0 in the system, so rank 1 is invalid.
>>>>> please remove "rank 1=node2 slot=*" from the rankfile
>>>>> Best regards,
>>>>> Lenny.
>>>>>
>>>>> On Mon, May 4, 2009 at 11:14 AM, Geoffroy Pignot <geopig...@gmail.com>
>>>>> wrote:
>>>>> Hi ,
>>>>>
>>>>> I got the openmpi-1.4a1r21095.tar.gz tarball, but unfortunately my
>>>>> command doesn't work
>>>>>
>>>>> cat rankf:
>>>>> rank 0=node1 slot=*
>>>>> rank 1=node2 slot=*
>>>>>
>>>>> cat hostf:
>>>>> node1 slots=2
>>>>> node2 slots=2
>>>>>
>>>>> mpirun --rankfile rankf --hostfile hostf --host node1 -n 1 hostname :
>>>>> --host node2 -n 1 hostname
>>>>>
>>>>> Error, invalid rank (1) in the rankfile (rankf)
>>>>>
>>>>>
>>>>> --------------------------------------------------------------------------
>>>>> [r011n006:28986] [[45541,0],0] ORTE_ERROR_LOG: Bad parameter in file
>>>>> rmaps_rank_file.c at line 403
>>>>> [r011n006:28986] [[45541,0],0] ORTE_ERROR_LOG: Bad parameter in file
>>>>> base/rmaps_base_map_job.c at line 86
>>>>> [r011n006:28986] [[45541,0],0] ORTE_ERROR_LOG: Bad parameter in file
>>>>> base/plm_base_launch_support.c at line 86
>>>>> [r011n006:28986] [[45541,0],0] ORTE_ERROR_LOG: Bad parameter in file
>>>>> plm_rsh_module.c at line 1016
>>>>>
>>>>>
>>>>> Ralph, could you tell me if my command syntax is correct or not ? if
>>>>> not, give me the expected one ?
>>>>>
>>>>> Regards
>>>>>
>>>>> Geoffroy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2009/4/30 Geoffroy Pignot <geopig...@gmail.com>
>>>>>
>>>>> Immediately Sir !!! :)
>>>>>
>>>>> Thanks again Ralph
>>>>>
>>>>> Geoffroy
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 2
>>>>> Date: Thu, 30 Apr 2009 06:45:39 -0600
>>>>> From: Ralph Castain <r...@open-mpi.org>
>>>>> Subject: Re: [OMPI users] 1.3.1 -rf rankfile behaviour ??
>>>>> To: Open MPI Users <us...@open-mpi.org>
>>>>> Message-ID:
>>>>> <71d2d8cc0904300545v61a42fe1k50086d2704d0f...@mail.gmail.com>
>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>
>>>>> I believe this is fixed now in our development trunk - you can download
>>>>> any
>>>>> tarball starting from last night and give it a try, if you like. Any
>>>>> feedback would be appreciated.
>>>>>
>>>>> Ralph
>>>>>
>>>>>
>>>>> On Apr 14, 2009, at 7:57 AM, Ralph Castain wrote:
>>>>>
>>>>> Ah now, I didn't say it -worked-, did I? :-)
>>>>>
>>>>> Clearly a bug exists in the program. I'll try to take a look at it (if
>>>>> Lenny
>>>>> doesn't get to it first), but it won't be until later in the week.
>>>>>
>>>>> On Apr 14, 2009, at 7:18 AM, Geoffroy Pignot wrote:
>>>>>
>>>>> I agree with you Ralph , and that 's what I expect from openmpi but my
>>>>> second example shows that it's not working
>>>>>
>>>>> cat hostfile.0
>>>>> r011n002 slots=4
>>>>> r011n003 slots=4
>>>>>
>>>>> cat rankfile.0
>>>>> rank 0=r011n002 slot=0
>>>>> rank 1=r011n003 slot=1
>>>>>
>>>>> mpirun --hostfile hostfile.0 -rf rankfile.0 -n 1 hostname : -n 1
>>>>> hostname
>>>>> ### CRASHED
>>>>>
>>>>> > > Error, invalid rank (1) in the rankfile (rankfile.0)
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > rmaps_rank_file.c at line 404
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > base/rmaps_base_map_job.c at line 87
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > base/plm_base_launch_support.c at line 77
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > plm_rsh_module.c at line 985
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > A daemon (pid unknown) died unexpectedly on signal 1 while
>>>>> > attempting to
>>>>> > > launch so we are aborting.
>>>>> > >
>>>>> > > There may be more information reported by the environment (see
>>>>> > above).
>>>>> > >
>>>>> > > This may be because the daemon was unable to find all the needed
>>>>> > shared
>>>>> > > libraries on the remote node. You may set your LD_LIBRARY_PATH to
>>>>> > have the
>>>>> > > location of the shared libraries on the remote nodes and this will
>>>>> > > automatically be forwarded to the remote nodes.
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > orterun noticed that the job aborted, but has no info as to the
>>>>> > process
>>>>> > > that caused that situation.
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > orterun: clean termination accomplished
>>>>>
>>>>>
>>>>>
>>>>> Message: 4
>>>>> Date: Tue, 14 Apr 2009 06:55:58 -0600
>>>>> From: Ralph Castain <r...@lanl.gov>
>>>>> Subject: Re: [OMPI users] 1.3.1 -rf rankfile behaviour ??
>>>>> To: Open MPI Users <us...@open-mpi.org>
>>>>> Message-ID: <f6290ada-a196-43f0-a853-cbcb802d8...@lanl.gov>
>>>>> Content-Type: text/plain; charset="us-ascii"; Format="flowed";
>>>>> DelSp="yes"
>>>>>
>>>>> The rankfile cuts across the entire job - it isn't applied on an
>>>>> app_context basis. So the ranks in your rankfile must correspond to
>>>>> the eventual rank of each process in the cmd line.
>>>>>
>>>>> Unfortunately, that means you have to count ranks. In your case, you
>>>>> only have four, so that makes life easier. Your rankfile would look
>>>>> something like this:
>>>>>
>>>>> rank 0=r001n001 slot=0
>>>>> rank 1=r001n002 slot=1
>>>>> rank 2=r001n001 slot=1
>>>>> rank 3=r001n002 slot=2
>>>>>
>>>>> HTH
>>>>> Ralph
>>>>>
>>>>> On Apr 14, 2009, at 12:19 AM, Geoffroy Pignot wrote:
>>>>>
>>>>> > Hi,
>>>>> >
>>>>> > I agree that my examples are not very clear. What I want to do is to
>>>>> > launch a multiexes application (masters-slaves) and benefit from the
>>>>> > processor affinity.
>>>>> > Could you show me how to convert this command , using -rf option
>>>>> > (whatever the affinity is)
>>>>> >
>>>>> > mpirun -n 1 -host r001n001 master.x options1 : -n 1 -host r001n002
>>>>> > master.x options2 : -n 1 -host r001n001 slave.x options3 : -n 1 -
>>>>> > host r001n002 slave.x options4
>>>>> >
>>>>> > Thanks for your help
>>>>> >
>>>>> > Geoffroy
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > Message: 2
>>>>> > Date: Sun, 12 Apr 2009 18:26:35 +0300
>>>>> > From: Lenny Verkhovsky <lenny.verkhov...@gmail.com>
>>>>> > Subject: Re: [OMPI users] 1.3.1 -rf rankfile behaviour ??
>>>>> > To: Open MPI Users <us...@open-mpi.org>
>>>>> > Message-ID:
>>>>> > <453d39990904120826t2e1d1d33l7bb1fe3de65b5...@mail.gmail.com>
>>>>> > Content-Type: text/plain; charset="iso-8859-1"
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > The first "crash" is OK, since your rankfile has ranks 0 and 1
>>>>> > defined,
>>>>> > while n=1, which means only rank 0 is present and can be allocated.
>>>>> >
>>>>> > NP must be >= the largest rank in rankfile.
>>>>> >
>>>>> > What exactly are you trying to do ?
>>>>> >
>>>>> > I tried to recreate your seqv but all I got was
>>>>> >
>>>>> > ~/work/svn/ompi/trunk/build_x86-64/install/bin/mpirun --hostfile
>>>>> > hostfile.0
>>>>> > -rf rankfile.0 -n 1 hostname : -rf rankfile.1 -n 1 hostname
>>>>> > [witch19:30798] mca: base: component_find: paffinity
>>>>> > "mca_paffinity_linux"
>>>>> > uses an MCA interface that is not recognized (component MCA v1.0.0 !=
>>>>> > supported MCA v2.0.0) -- ignored
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > It looks like opal_init failed for some reason; your parallel
>>>>> > process is
>>>>> > likely to abort. There are many reasons that a parallel process can
>>>>> > fail during opal_init; some of which are due to configuration or
>>>>> > environment problems. This failure appears to be an internal failure;
>>>>> > here's some additional information (which may only be relevant to an
>>>>> > Open MPI developer):
>>>>> >
>>>>> > opal_carto_base_select failed
>>>>> > --> Returned value -13 instead of OPAL_SUCCESS
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > [witch19:30798] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file
>>>>> > ../../orte/runtime/orte_init.c at line 78
>>>>> > [witch19:30798] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file
>>>>> > ../../orte/orted/orted_main.c at line 344
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > A daemon (pid 11629) died unexpectedly with status 243 while
>>>>> > attempting
>>>>> > to launch so we are aborting.
>>>>> >
>>>>> > There may be more information reported by the environment (see
>>>>> above).
>>>>> >
>>>>> > This may be because the daemon was unable to find all the needed
>>>>> > shared
>>>>> > libraries on the remote node. You may set your LD_LIBRARY_PATH to
>>>>> > have the
>>>>> > location of the shared libraries on the remote nodes and this will
>>>>> > automatically be forwarded to the remote nodes.
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > mpirun noticed that the job aborted, but has no info as to the
>>>>> process
>>>>> > that caused that situation.
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > mpirun: clean termination accomplished
>>>>> >
>>>>> >
>>>>> > Lenny.
>>>>> >
>>>>> >
>>>>> > On 4/10/09, Geoffroy Pignot <geopig...@gmail.com> wrote:
>>>>> > >
>>>>> > > Hi ,
>>>>> > >
>>>>> > > I am currently testing the process affinity capabilities of
>>>>> > openmpi and I
>>>>> > > would like to know if the rankfile behaviour I will describe below
>>>>> > is normal
>>>>> > > or not ?
>>>>> > >
>>>>> > > cat hostfile.0
>>>>> > > r011n002 slots=4
>>>>> > > r011n003 slots=4
>>>>> > >
>>>>> > > cat rankfile.0
>>>>> > > rank 0=r011n002 slot=0
>>>>> > > rank 1=r011n003 slot=1
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>> ##################################################################################
>>>>> > >
>>>>> > > mpirun --hostfile hostfile.0 -rf rankfile.0 -n 2 hostname ### OK
>>>>> > > r011n002
>>>>> > > r011n003
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>> ##################################################################################
>>>>> > > but
>>>>> > > mpirun --hostfile hostfile.0 -rf rankfile.0 -n 1 hostname : -n 1
>>>>> > hostname
>>>>> > > ### CRASHED
>>>>> > > *
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > Error, invalid rank (1) in the rankfile (rankfile.0)
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > rmaps_rank_file.c at line 404
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > base/rmaps_base_map_job.c at line 87
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > base/plm_base_launch_support.c at line 77
>>>>> > > [r011n002:25129] [[63976,0],0] ORTE_ERROR_LOG: Bad parameter in
>>>>> file
>>>>> > > plm_rsh_module.c at line 985
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > A daemon (pid unknown) died unexpectedly on signal 1 while
>>>>> > attempting to
>>>>> > > launch so we are aborting.
>>>>> > >
>>>>> > > There may be more information reported by the environment (see
>>>>> > above).
>>>>> > >
>>>>> > > This may be because the daemon was unable to find all the needed
>>>>> > shared
>>>>> > > libraries on the remote node. You may set your LD_LIBRARY_PATH to
>>>>> > have the
>>>>> > > location of the shared libraries on the remote nodes and this will
>>>>> > > automatically be forwarded to the remote nodes.
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > orterun noticed that the job aborted, but has no info as to the
>>>>> > process
>>>>> > > that caused that situation.
>>>>> > >
>>>>> >
>>>>> --------------------------------------------------------------------------
>>>>> > > orterun: clean termination accomplished
>>>>> > > *
>>>>> > > It seems that the rankfile option is not propagted to the second
>>>>> > command
>>>>> > > line ; there is no global understanding of the ranking inside a
>>>>> > mpirun
>>>>> > > command.
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>> ##################################################################################
>>>>> > >
>>>>> > > Assuming that , I tried to provide a rankfile to each command line:
>>>>> > >
>>>>> > > cat rankfile.0
>>>>> > > rank 0=r011n002 slot=0
>>>>> > >
>>>>> > > cat rankfile.1
>>>>> > > rank 0=r011n003 slot=1
>>>>> > >
>>>>> > > mpirun --hostfile hostfile.0 -rf rankfile.0 -n 1 hostname : -rf
>>>>> > rankfile.1
>>>>> > > -n 1 hostname ### CRASHED
>>>>> > > *[r011n002:28778] *** Process received signal ***
>>>>> > > [r011n002:28778] Signal: Segmentation fault (11)
>>>>> > > [r011n002:28778] Signal code: Address not mapped (1)
>>>>> > > [r011n002:28778] Failing at address: 0x34
>>>>> > > [r011n002:28778] [ 0] [0xffffe600]
>>>>> > > [r011n002:28778] [ 1]
>>>>> > > /tmp/HALMPI/openmpi-1.3.1/lib/libopen-rte.so.
>>>>> > 0(orte_odls_base_default_get_add_procs_data+0x55d)
>>>>> > > [0x5557decd]
>>>>> > > [r011n002:28778] [ 2]
>>>>> > > /tmp/HALMPI/openmpi-1.3.1/lib/libopen-rte.so.
>>>>> > 0(orte_plm_base_launch_apps+0x117)
>>>>> > > [0x555842a7]
>>>>> > > [r011n002:28778] [ 3] /tmp/HALMPI/openmpi-1.3.1/lib/openmpi/
>>>>> > mca_plm_rsh.so
>>>>> > > [0x556098c0]
>>>>> > > [r011n002:28778] [ 4] /tmp/HALMPI/openmpi-1.3.1/bin/orterun
>>>>> > [0x804aa27]
>>>>> > > [r011n002:28778] [ 5] /tmp/HALMPI/openmpi-1.3.1/bin/orterun
>>>>> > [0x804a022]
>>>>> > > [r011n002:28778] [ 6] /lib/libc.so.6(__libc_start_main+0xdc)
>>>>> > [0x9f1dec]
>>>>> > > [r011n002:28778] [ 7] /tmp/HALMPI/openmpi-1.3.1/bin/orterun
>>>>> > [0x8049f71]
>>>>> > > [r011n002:28778] *** End of error message ***
>>>>> > > Segmentation fault (core dumped)*
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > I hope that I've found a bug because it would be very important
>>>>> > for me to
>>>>> > > have this kind of capabiliy .
>>>>> > > Launch a multiexe mpirun command line and be able to bind my exes
>>>>> > and
>>>>> > > sockets together.
>>>>> > >
>>>>> > > Thanks in advance for your help
>>>>> > >
>>>>> > > Geoffroy
>>>>> > _______________________________________________
>>>>> > users mailing list
>>>>> > us...@open-mpi.org
>>>>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> -------------- next part --------------
>>>>> HTML attachment scrubbed and removed
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> End of users Digest, Vol 1202, Issue 2
>>>>> **************************************
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> -------------- next part --------------
>>>>> HTML attachment scrubbed and removed
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> End of users Digest, Vol 1218, Issue 2
>>>>> **************************************
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
