Ralph,
Sorry, "<" as an element of argv to mpi_comm_spawn is interpreted just the
same, as another parameter by the spawnee process.
But I am confused: wouldn't it be redundant to put "mpirun" "-n" "1"
"myapp"
as elements of argv, considering role of the other parameters of
mpi_comm_spawn
like the 1st and 3rd ?
Imho, it seems to me that stdin and stdout should be keys to "info" and the
respective filenames the values of those keys, if that is possible to
implement...
Alex
2014-12-17 15:16 GMT-02:00 Ralph Castain <r...@open-mpi.org>:
>
> Have you tried putting the "<" as a separate parameter? In other words,
> since you are specifying the argv, you have to specify each of them
> separately. So it looks more like:
>
> "mpirun", "-n", "1", "myapp", "<", "stdinfile"
>
> Does that work?
> Ralph
>
>
> On Wed, Dec 17, 2014 at 8:07 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
>
>> Ralph,
>>
>> I am afraid I will have to insist on i/o redirection matter
>> for the spawnee process.
>>
>> I have a "child" mpi code that do just 2 things: read the 3 parameters
>> passed to it and print them, and then read data from stdin and show it.
>> So, if "stdin_file" is a text file with two lines, say:
>>
>> 10
>> 20
>>
>> executing "mpirun -n 1 child A B < stdin_file" wiil ouput two lines:
>>
>> [A] [B] []
>> 10 20
>>
>> On the other hand , calling "child" from MPI_Comm_spawn("child",args,...)
>> where
>>
>> args(1) = "A"
>> args(2) = "B"
>> args(3) ="< stdin_file"
>> args(4) = " "
>>
>> will make "child" outputs only 1 line
>>
>> [A] [B] [< stdin_file]
>>
>> and then fails because there is not stdin data to read from.
>>
>> Please, note that surprisingly the whole string "< stdin_file" is
>> interpreted
>> as a third parameter to "child" and not a stdin...
>>
>> Alex
>>
>>
>>
>>
>>
>>
>>
>> 2014-12-15 17:26 GMT-02:00 Alex A. Schmidt <a...@ufsm.br>:
>>
>>> Ralph,
>>>
>>> I guess you mean "call mpi_comm_spawn( 'siesta', '< infile' , 2 ,...)"
>>>
>>> to execute 'mpirun -n 2 siesta < infile' on the spawnee side. That was
>>> my first choice. Well, siesta behaves as if no stdin file was present...
>>>
>>> Alex
>>>
>>>
>>> 2014-12-15 17:07 GMT-02:00 Ralph Castain <r...@open-mpi.org>:
>>>>
>>>> You should be able to just include that in your argv that you pass to
>>>> the Comm_spawn API.
>>>>
>>>>
>>>> On Mon, Dec 15, 2014 at 9:27 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
>>>>
>>>>> George,
>>>>>
>>>>> Thanks for the tip. In fact, calling mpi_comm_spawn right away with
>>>>> MPI_COMM_SELF
>>>>> has worked for me just as well -- no subgroups needed at all.
>>>>>
>>>>> I am testing this openmpi app named "siesta" in parallel. The source
>>>>> code is available,
>>>>> so making it "spawn ready" by adding the pair mpi_comm_get_parent +
>>>>> mpi_comm_disconnect
>>>>> into the main code can be done. If it works, maybe the siesta's
>>>>> developers can be convinced
>>>>> to add this feature in a future release.
>>>>>
>>>>> However, siesta is launched only by specifying input/output files
>>>>> with i/o redirection like
>>>>>
>>>>> mpirun -n <*some number*> siesta < infile > outfile
>>>>>
>>>>> So far, I could not find anything about how to set an stdin file for
>>>>> an spawnee process.
>>>>> Specifiyng it in a app context file doesn't seem to work. Can it be
>>>>> done? Maybe through
>>>>> an MCA parameter?
>>>>>
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2014-12-15 2:43 GMT-02:00 George Bosilca <bosi...@icl.utk.edu>:
>>>>>>
>>>>>> Alex,
>>>>>>
>>>>>> The code looks good, and is 100% MPI standard accurate.
>>>>>>
>>>>>> I would change the way you create the subcoms in the parent. You do a
>>>>>> lot of useless operations, as you can achieve exactly the same outcome
>>>>>> (one
>>>>>> communicator per node), either by duplicating MPI_COMM_SELF or doing an
>>>>>> MPI_Comm_split with the color equal to your rank.
>>>>>>
>>>>>> George.
>>>>>>
>>>>>>
>>>>>> On Sun, Dec 14, 2014 at 2:20 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Sorry, guys. I don't think the newbie here can follow any discussion
>>>>>>> beyond basic mpi...
>>>>>>>
>>>>>>> Anyway, if I add the pair
>>>>>>>
>>>>>>> call MPI_COMM_GET_PARENT(mpi_comm_parent,ierror)
>>>>>>> call MPI_COMM_DISCONNECT(mpi_comm_parent,ierror)
>>>>>>>
>>>>>>> on the spawnee side I get the proper response in the spawning
>>>>>>> processes.
>>>>>>>
>>>>>>> Please, take a look at the attached toy codes parent.F and child.F
>>>>>>> I've been playing with. 'mpirun -n 2 parent' seems to work as
>>>>>>> expected.
>>>>>>>
>>>>>>> Alex
>>>>>>>
>>>>>>> 2014-12-13 23:46 GMT-02:00 Gilles Gouaillardet <
>>>>>>> gilles.gouaillar...@gmail.com>:
>>>>>>>>
>>>>>>>> Alex,
>>>>>>>>
>>>>>>>> Are you calling MPI_Comm_disconnect in the 3 "master" tasks and
>>>>>>>> with the same remote communicator ?
>>>>>>>>
>>>>>>>> I also read the man page again, and MPI_Comm_disconnect does not
>>>>>>>> ensure the remote processes have finished or called
>>>>>>>> MPI_Comm_disconnect, so
>>>>>>>> that might not be the thing you need.
>>>>>>>> George, can you please comment on that ?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Gilles
>>>>>>>>
>>>>>>>> George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>>>>> MPI_Comm_disconnect should be a local operation, there is no reason
>>>>>>>> for it to deadlock. I looked at the code and everything is local with
>>>>>>>> the
>>>>>>>> exception of a call to PMIX.FENCE. Can you attach to your deadlocked
>>>>>>>> processes and confirm that they are stopped in the pmix.fence?
>>>>>>>>
>>>>>>>> George.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Dec 13, 2014 at 8:47 AM, Alex A. Schmidt <a...@ufsm.br>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> Sorry, I was calling mpi_comm_disconnect on the group comm
>>>>>>>>> handler, not
>>>>>>>>> on the intercomm handler returned from the spawn call as it should
>>>>>>>>> be.
>>>>>>>>>
>>>>>>>>> Well, calling the disconnect on the intercomm handler does halt
>>>>>>>>> the spwaner
>>>>>>>>> side but the wait is never completed since, as George points out,
>>>>>>>>> there is no
>>>>>>>>> disconnect call being made on the spawnee side.... and that brings
>>>>>>>>> me back
>>>>>>>>> to the beginning of the problem since, being a third party app,
>>>>>>>>> that call would
>>>>>>>>> never be there. I guess an mpi wrapper to deal with that could be
>>>>>>>>> made for
>>>>>>>>> the app, but I fell the wrapper itself, at the end, would face the
>>>>>>>>> same problem
>>>>>>>>> we face right now.
>>>>>>>>>
>>>>>>>>> My application is a genetic algorithm code that search optimal
>>>>>>>>> configuration
>>>>>>>>> (minimum or maximum energy) of cluster of atoms. The work flow
>>>>>>>>> bottleneck
>>>>>>>>> is the calculation of the cluster energy. For the cases which an
>>>>>>>>> analytical
>>>>>>>>> potential is available the calculation can be made internally and
>>>>>>>>> the workload
>>>>>>>>> is distributed among slaves nodes from a master node. This is also
>>>>>>>>> done
>>>>>>>>> when an analytical potential is not available and the energy
>>>>>>>>> calculation must
>>>>>>>>> be done externally by a quantum chemistry code like dftb+, siesta
>>>>>>>>> and Gaussian.
>>>>>>>>> So far, we have been running these codes in serial mode. No need
>>>>>>>>> to say that
>>>>>>>>> we could do a lot better if they could be executed in parallel.
>>>>>>>>>
>>>>>>>>> I am not familiar with DMRAA but it seems to be the right choice
>>>>>>>>> to deal with
>>>>>>>>> job schedulers as it covers the ones I am interested in
>>>>>>>>> (pbs/torque and loadlever).
>>>>>>>>>
>>>>>>>>> Alex
>>>>>>>>>
>>>>>>>>> 2014-12-13 7:49 GMT-02:00 Gilles Gouaillardet <
>>>>>>>>> gilles.gouaillar...@gmail.com>:
>>>>>>>>>>
>>>>>>>>>> George is right about the semantic
>>>>>>>>>>
>>>>>>>>>> However i am surprised it returns immediatly...
>>>>>>>>>> That should either work or hang imho
>>>>>>>>>>
>>>>>>>>>> The second point is no more mpi related, and is batch manager
>>>>>>>>>> specific.
>>>>>>>>>>
>>>>>>>>>> You will likely find a submit parameter to make the command block
>>>>>>>>>> until the job completes. Or you can write your own wrapper.
>>>>>>>>>> Or you can retrieve the jobid and qstat periodically to get the
>>>>>>>>>> job state.
>>>>>>>>>> If an api is available, this is also an option.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Gilles
>>>>>>>>>>
>>>>>>>>>> George Bosilca <bosi...@icl.utk.edu> wrote:
>>>>>>>>>> You have to call MPI_Comm_disconnect on both sides of the
>>>>>>>>>> intercommunicator. On the spawner processes you should call it on the
>>>>>>>>>> intercom, while on the spawnees you should call it on the
>>>>>>>>>> MPI_Comm_get_parent.
>>>>>>>>>>
>>>>>>>>>> George.
>>>>>>>>>>
>>>>>>>>>> On Dec 12, 2014, at 20:43 , Alex A. Schmidt <a...@ufsm.br> wrote:
>>>>>>>>>>
>>>>>>>>>> Gilles,
>>>>>>>>>>
>>>>>>>>>> MPI_comm_disconnect seem to work but not quite.
>>>>>>>>>> The call to it returns almost immediatly while
>>>>>>>>>> the spawn processes keep piling up in the background
>>>>>>>>>> until they are all done...
>>>>>>>>>>
>>>>>>>>>> I think system('env -i qsub...') to launch the third party apps
>>>>>>>>>> would take the execution of every call back to the scheduler
>>>>>>>>>> queue. How would I track each one for their completion?
>>>>>>>>>>
>>>>>>>>>> Alex
>>>>>>>>>>
>>>>>>>>>> 2014-12-12 22:35 GMT-02:00 Gilles Gouaillardet <
>>>>>>>>>> gilles.gouaillar...@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>> Alex,
>>>>>>>>>>>
>>>>>>>>>>> You need MPI_Comm_disconnect at least.
>>>>>>>>>>> I am not sure if this is 100% correct nor working.
>>>>>>>>>>>
>>>>>>>>>>> If you are using third party apps, why dont you do something like
>>>>>>>>>>> system("env -i qsub ...")
>>>>>>>>>>> with the right options to make qsub blocking or you manually
>>>>>>>>>>> wait for the end of the job ?
>>>>>>>>>>>
>>>>>>>>>>> That looks like a much cleaner and simpler approach to me.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Gilles
>>>>>>>>>>>
>>>>>>>>>>> "Alex A. Schmidt" <a...@ufsm.br> wrote:
>>>>>>>>>>> Hello Gilles,
>>>>>>>>>>>
>>>>>>>>>>> Ok, I believe I have a simple toy app running as I think it
>>>>>>>>>>> should:
>>>>>>>>>>> 'n' parent processes running under mpi_comm_world, each one
>>>>>>>>>>> spawning its own 'm' child processes (each child group work
>>>>>>>>>>> together nicely, returning the expected result for an mpi_
>>>>>>>>>>> allreduce call).
>>>>>>>>>>>
>>>>>>>>>>> Now, as I mentioned before, the apps I want to run in the
>>>>>>>>>>> spawned
>>>>>>>>>>> processes are third party mpi apps and I don't think it will be
>>>>>>>>>>> possible
>>>>>>>>>>> to exchange messages with them from my app. So, I do I tell
>>>>>>>>>>> when the spawned processes have finnished running? All I have to
>>>>>>>>>>> work
>>>>>>>>>>> with is the intercommunicator returned from the mpi_comm_spawn
>>>>>>>>>>> call...
>>>>>>>>>>>
>>>>>>>>>>> Alex
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2014-12-12 2:42 GMT-02:00 Alex A. Schmidt <a...@ufsm.br>:
>>>>>>>>>>>>
>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>
>>>>>>>>>>>> Well, yes, I guess....
>>>>>>>>>>>>
>>>>>>>>>>>> I'll do tests with the real third party apps and let you know.
>>>>>>>>>>>> These are huge quantum chemistry codes (dftb+, siesta and
>>>>>>>>>>>> Gaussian)
>>>>>>>>>>>> which greatly benefits from a parallel environment. My code is
>>>>>>>>>>>> just
>>>>>>>>>>>> a front end to use those, but since we have a lot of data to
>>>>>>>>>>>> process
>>>>>>>>>>>> it also benefits from a parallel environment.
>>>>>>>>>>>>
>>>>>>>>>>>> Alex
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2014-12-12 2:30 GMT-02:00 Gilles Gouaillardet <
>>>>>>>>>>>> gilles.gouaillar...@iferc.org>:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex,
>>>>>>>>>>>>>
>>>>>>>>>>>>> just to make sure ...
>>>>>>>>>>>>> this is the behavior you expected, right ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gilles
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2014/12/12 13:27, Alex A. Schmidt wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gilles,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, very nice!
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I excute
>>>>>>>>>>>>>
>>>>>>>>>>>>> do rank=1,3
>>>>>>>>>>>>> call MPI_Comm_spawn('hello_world','
>>>>>>>>>>>>> ',5,MPI_INFO_NULL,rank,MPI_COMM_WORLD,my_intercomm,MPI_ERRCODES_IGNORE,status)
>>>>>>>>>>>>> enddo
>>>>>>>>>>>>>
>>>>>>>>>>>>> I do get 15 instances of the 'hello_world' app running: 5 for
>>>>>>>>>>>>> each parent
>>>>>>>>>>>>> rank 1, 2 and 3.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks a lot, Gilles.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regargs,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2014-12-12 1:32 GMT-02:00 Gilles Gouaillardet
>>>>>>>>>>>>> <gilles.gouaillar...@iferc.org
>>>>>>>>>>>>>
>>>>>>>>>>>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex,
>>>>>>>>>>>>>
>>>>>>>>>>>>> just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs
>>>>>>>>>>>>> parameter :
>>>>>>>>>>>>>
>>>>>>>>>>>>> int MPI_Comm_spawn(char *command, char *argv[], int
>>>>>>>>>>>>> maxprocs,
>>>>>>>>>>>>> MPI_Info info,
>>>>>>>>>>>>> int root, MPI_Comm comm, MPI_Comm
>>>>>>>>>>>>> *intercomm,
>>>>>>>>>>>>> int array_of_errcodes[])
>>>>>>>>>>>>>
>>>>>>>>>>>>> INPUT PARAMETERS
>>>>>>>>>>>>> maxprocs
>>>>>>>>>>>>> - maximum number of processes to start (integer,
>>>>>>>>>>>>> significant
>>>>>>>>>>>>> only at root)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gilles
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2014/12/12 12:23, Alex A. Schmidt wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello Gilles,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your reply. The "env -i PATH=..." stuff seems to
>>>>>>>>>>>>> work!!!
>>>>>>>>>>>>>
>>>>>>>>>>>>> call system("sh -c 'env -i PATH=/usr/lib64/openmpi/bin:/bin
>>>>>>>>>>>>> mpirun -n 2
>>>>>>>>>>>>> hello_world' ")
>>>>>>>>>>>>>
>>>>>>>>>>>>> did produce the expected result with a simple openmi
>>>>>>>>>>>>> "hello_world" code I
>>>>>>>>>>>>> wrote.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I might be harder though with the real third party app I have in
>>>>>>>>>>>>> mind. And
>>>>>>>>>>>>> I realize
>>>>>>>>>>>>> getting passed over a job scheduler with this approach might not
>>>>>>>>>>>>> work at
>>>>>>>>>>>>> all...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have looked at the MPI_Comm_spawn call but I failed to
>>>>>>>>>>>>> understand how it
>>>>>>>>>>>>> could help here. For instance, can I use it to launch an mpi app
>>>>>>>>>>>>> with the
>>>>>>>>>>>>> option "-n 5" ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2014-12-12 0:36 GMT-02:00 Gilles Gouaillardet
>>>>>>>>>>>>> <gilles.gouaillar...@iferc.org
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex,
>>>>>>>>>>>>>
>>>>>>>>>>>>> can you try something like
>>>>>>>>>>>>> call system(sh -c 'env -i /.../mpirun -np 2 /.../app_name')
>>>>>>>>>>>>>
>>>>>>>>>>>>> -i start with an empty environment
>>>>>>>>>>>>> that being said, you might need to set a few environment variables
>>>>>>>>>>>>> manually :
>>>>>>>>>>>>> env -i PATH=/bin ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> and that being also said, this "trick" could be just a bad idea :
>>>>>>>>>>>>> you might be using a scheduler, and if you empty the environment,
>>>>>>>>>>>>> the
>>>>>>>>>>>>> scheduler
>>>>>>>>>>>>> will not be aware of the "inside" run.
>>>>>>>>>>>>>
>>>>>>>>>>>>> on top of that, invoking system might fail depending on the
>>>>>>>>>>>>> interconnect
>>>>>>>>>>>>> you use.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bottom line, i believe Ralph's reply is still valid, even if five
>>>>>>>>>>>>> years
>>>>>>>>>>>>> have passed :
>>>>>>>>>>>>> changing your workflow, or using MPI_Comm_spawn is a much better
>>>>>>>>>>>>> approach.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gilles
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2014/12/12 11:22, Alex A. Schmidt wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Dear OpenMPI users,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding to this previous
>>>>>>>>>>>>> post<http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>>>>>>>>>>> from 2009,
>>>>>>>>>>>>> I wonder if the reply
>>>>>>>>>>>>> from Ralph Castain is still valid. My need is similar but quite
>>>>>>>>>>>>> simpler:
>>>>>>>>>>>>> to make a system call from an openmpi fortran application to run a
>>>>>>>>>>>>> third party openmpi application. I don't need to exchange mpi
>>>>>>>>>>>>> messages
>>>>>>>>>>>>> with the application. I just need to read the resulting output
>>>>>>>>>>>>> file
>>>>>>>>>>>>> generated
>>>>>>>>>>>>> by it. I have tried to do the following system call from my
>>>>>>>>>>>>> fortran openmpi
>>>>>>>>>>>>> code:
>>>>>>>>>>>>>
>>>>>>>>>>>>> call system("sh -c 'mpirun -n 2 app_name")
>>>>>>>>>>>>>
>>>>>>>>>>>>> but I get
>>>>>>>>>>>>>
>>>>>>>>>>>>> **********************************************************
>>>>>>>>>>>>>
>>>>>>>>>>>>> Open MPI does not support recursive calls of mpirun
>>>>>>>>>>>>>
>>>>>>>>>>>>> **********************************************************
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there a way to make this work?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alex
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing listus...@open-mpi.org
>>>>>>>>>>>>>
>>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> Link to this post:
>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25966.php
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing listus...@open-mpi.org
>>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> Link to this
>>>>>>>>>>>>> post:http://www.open-mpi.org/community/lists/users/2014/12/25967.php
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing listus...@open-mpi.org
>>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>
>>>>>>>>>>>>> Link to this post:
>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25968.php
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing listus...@open-mpi.org
>>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> Link to this
>>>>>>>>>>>>> post:http://www.open-mpi.org/community/lists/users/2014/12/25969.php
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing listus...@open-mpi.org
>>>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> Link to this post:
>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25970.php
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>> Subscription:
>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>> Link to this post:
>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25971.php
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>> Link to this post:
>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25974.php
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>> Link to this post:
>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25975.php
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>> Link to this post:
>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25978.php
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> us...@open-mpi.org
>>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>> Link to this post:
>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25979.php
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org
>>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>> Link to this post:
>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25981.php
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25982.php
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/users/2014/12/25991.php
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/users/2014/12/26002.php
>>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2014/12/26003.php
>>>>
>>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/12/26010.php
>>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26011.php
>
