Am 18.12.2014 um 04:24 schrieb Alex A. Schmidt <a...@ufsm.br>:
>
> The option system("env -i ...") has been tested earlier by me and it does
> work. There is doubt though it would work along with a job scheduler.
> I will reserve this as a last resort solution.
You could also redirect the stdin before the `system()` call in your
application by dup2(). But I'm somewhat lost in the complete setup: you have a
batch scheduler, your MPI application and third party apps (using MPI, Linda,
OpenMP,...) with both of the latter working in parallel (how to forward granted
machines/cores to siesta in case it should spread further to other machines?).
What will call what? Do you submit your application or the third party apps as
mentioned earlier? Will the batch scheduler honor any redirection which you
discussed recently, and/or you can you tell the batch scheduler to use a
different stdin/-out/-err in DRMAA by setting
drmaa_input_path/drmaa_output_path/drmaa_error_path for example?
-- Reuti
> mpi_comm_spawn("/bin/sh","-c","siesta < infile",..) definitely does not work.
>
> Patching siesta to start as "siesta -in infile" sounds very promissing.
> That option as well as a "-o outfile" option should be there to start with.
> I can bring that to the attention of the siesta developers. But then we have
> this other app "dftb+", and this other one "Gaussian" (just to mention a few).
> Some of them might already act like that, others don't. So, having
> mpi_comm_spawn working with i/o redirection would be in fact more
> simpler.
>
> Perhaps a shell script "siesta_alt" could be written to convert
> "siesta_alt infile outfile" into "siesta < infile > outfile" and then
> do mpi_comm_spawn("siesta_alt","infile","outfile",...). But I am not
> sure if the spawn on siesta_alt would actually lead to a spawn on siesta
> as expected...
>
> Alex
>
> 2014-12-17 22:35 GMT-02:00 Gilles Gouaillardet
> <gilles.gouaillar...@gmail.com>:
> Alex,
>
> You do not want to spawn mpirun.
> Or if this is really what you want, then just use system("env -i ...")
>
> I think what you need is spawn a shell that do the redirection and then
> invoke your app.
> This is something like
> MPI_Comm_spawn("/bin/sh", "-c", "siesta < infile")
>
> That being said, i strongly recommend you patch siesta so it can be invoked
> like this
> siesta -in infile
> (plus the MPI_Comm_disconnect call explained by George)
> That would make everything so much easier
>
> Cheers,
>
> Gilles
>
> "Alex A. Schmidt" <a...@ufsm.br> wrote:
> Let me rephrase the previous message:
>
> Putting "/bin/sh" in command with info key "ompi_non_mpi" set to ".true."
> (if command is empty, mpi_comm_spawn tries to execute ' ') of
> mpi_comm_spawn and "-c" "mpirun -n 1 myapp" in args results in
> this message:
>
> **********************************************************
>
> Open MPI does not support recursive calls of mpirun
>
> **********************************************************
>
> Putting a single string in args as "-c mpirun -n 1 myapp" or "-c 'mpirun -n
> 1 myapp' "
> returns
>
> /usr/bin/sh: - : invalid option
>
> Alex
>
>
>
> 2014-12-17 21:47 GMT-02:00 Alex A. Schmidt <a...@ufsm.br>:
> Putting "/bin/sh" in command with info key "ompi_non_mpi" set to ".true."
> (if command is empty, mpi_comm_spawn tries to execute ' ') of
> mpi_comm_spawn and "-c" "mpirun -n 1 myapp" in args results in
> this message:
>
> /usr/bin/sh: -c: option requires an argument
>
> Putting a single string in args as "-c mpirun -n 1 myapp" or "-c 'mpirun -n
> 1 myapp' "
> returns
>
> /usr/bin/sh: - : invalid option
>
> Alex
>
> 2014-12-17 20:17 GMT-02:00 George Bosilca <bosi...@icl.utk.edu>:
> I don't think this has any chance of working. The redirection is something
> interpreted by the shell, and when Open MPI "fork-exec" a process it does not
> behave as the shell.
>
> Thus a potentially non-portable solution would be to instead of launching the
> mpirun directly to launch it through a shell. Maybe something like "/bin/sh",
> "-c", "mpirun -n 1 myapp".
>
> George.
>
>
> On Wed, Dec 17, 2014 at 5:02 PM, Alex A. Schmidt <a...@ufsm.br> wrote:
> Ralph,
>
> Sorry, "<" as an element of argv to mpi_comm_spawn is interpreted just the
> same, as another parameter by the spawnee process.
>
> But I am confused: wouldn't it be redundant to put "mpirun" "-n" "1" "myapp"
> as elements of argv, considering role of the other parameters of
> mpi_comm_spawn
> like the 1st and 3rd ?
>
> Imho, it seems to me that stdin and stdout should be keys to "info" and the
> respective filenames the values of those keys, if that is possible to
> implement...
>
> Alex
>
>
>
> 2014-12-17 15:16 GMT-02:00 Ralph Castain <r...@open-mpi.org>:
> Have you tried putting the "<" as a separate parameter? In other words, since
> you are specifying the argv, you have to specify each of them separately. So
> it looks more like:
>
> "mpirun", "-n", "1", "myapp", "<", "stdinfile"
>
> Does that work?
> Ralph
>
>
> On Wed, Dec 17, 2014 at 8:07 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
> Ralph,
>
> I am afraid I will have to insist on i/o redirection matter
> for the spawnee process.
>
> I have a "child" mpi code that do just 2 things: read the 3 parameters
> passed to it and print them, and then read data from stdin and show it.
> So, if "stdin_file" is a text file with two lines, say:
>
> 10
> 20
>
> executing "mpirun -n 1 child A B < stdin_file" wiil ouput two lines:
>
> [A] [B] []
> 10 20
>
> On the other hand , calling "child" from MPI_Comm_spawn("child",args,...)
> where
>
> args(1) = "A"
> args(2) = "B"
> args(3) ="< stdin_file"
> args(4) = " "
>
> will make "child" outputs only 1 line
>
> [A] [B] [< stdin_file]
>
> and then fails because there is not stdin data to read from.
>
> Please, note that surprisingly the whole string "< stdin_file" is interpreted
> as a third parameter to "child" and not a stdin...
>
> Alex
>
>
>
>
>
>
>
> 2014-12-15 17:26 GMT-02:00 Alex A. Schmidt <a...@ufsm.br>:
> Ralph,
>
> I guess you mean "call mpi_comm_spawn( 'siesta', '< infile' , 2 ,...)"
>
> to execute 'mpirun -n 2 siesta < infile' on the spawnee side. That was
> my first choice. Well, siesta behaves as if no stdin file was present...
>
> Alex
>
>
> 2014-12-15 17:07 GMT-02:00 Ralph Castain <r...@open-mpi.org>:
> You should be able to just include that in your argv that you pass to the
> Comm_spawn API.
>
>
> On Mon, Dec 15, 2014 at 9:27 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
> George,
>
> Thanks for the tip. In fact, calling mpi_comm_spawn right away with
> MPI_COMM_SELF
> has worked for me just as well -- no subgroups needed at all.
>
> I am testing this openmpi app named "siesta" in parallel. The source code is
> available,
> so making it "spawn ready" by adding the pair mpi_comm_get_parent +
> mpi_comm_disconnect
> into the main code can be done. If it works, maybe the siesta's developers
> can be convinced
> to add this feature in a future release.
>
> However, siesta is launched only by specifying input/output files with i/o
> redirection like
>
> mpirun -n <some number> siesta < infile > outfile
>
> So far, I could not find anything about how to set an stdin file for an
> spawnee process.
> Specifiyng it in a app context file doesn't seem to work. Can it be done?
> Maybe through
> an MCA parameter?
>
> Alex
>
>
>
>
>
> 2014-12-15 2:43 GMT-02:00 George Bosilca <bosi...@icl.utk.edu>:
> Alex,
>
> The code looks good, and is 100% MPI standard accurate.
>
> I would change the way you create the subcoms in the parent. You do a lot of
> useless operations, as you can achieve exactly the same outcome (one
> communicator per node), either by duplicating MPI_COMM_SELF or doing an
> MPI_Comm_split with the color equal to your rank.
>
> George.
>
>
> On Sun, Dec 14, 2014 at 2:20 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
> Hi,
>
> Sorry, guys. I don't think the newbie here can follow any discussion
> beyond basic mpi...
>
> Anyway, if I add the pair
>
> call MPI_COMM_GET_PARENT(mpi_comm_parent,ierror)
> call MPI_COMM_DISCONNECT(mpi_comm_parent,ierror)
>
> on the spawnee side I get the proper response in the spawning processes.
>
> Please, take a look at the attached toy codes parent.F and child.F
> I've been playing with. 'mpirun -n 2 parent' seems to work as expected.
>
> Alex
>
> 2014-12-13 23:46 GMT-02:00 Gilles Gouaillardet
> <gilles.gouaillar...@gmail.com>:
> Alex,
>
> Are you calling MPI_Comm_disconnect in the 3 "master" tasks and with the same
> remote communicator ?
>
> I also read the man page again, and MPI_Comm_disconnect does not ensure the
> remote processes have finished or called MPI_Comm_disconnect, so that might
> not be the thing you need.
> George, can you please comment on that ?
>
> Cheers,
>
> Gilles
>
> George Bosilca <bosi...@icl.utk.edu> wrote:
> MPI_Comm_disconnect should be a local operation, there is no reason for it to
> deadlock. I looked at the code and everything is local with the exception of
> a call to PMIX.FENCE. Can you attach to your deadlocked processes and confirm
> that they are stopped in the pmix.fence?
>
> George.
>
>
> On Sat, Dec 13, 2014 at 8:47 AM, Alex A. Schmidt <a...@ufsm.br> wrote:
> Hi
>
> Sorry, I was calling mpi_comm_disconnect on the group comm handler, not
> on the intercomm handler returned from the spawn call as it should be.
>
> Well, calling the disconnect on the intercomm handler does halt the spwaner
> side but the wait is never completed since, as George points out, there is no
> disconnect call being made on the spawnee side.... and that brings me back
> to the beginning of the problem since, being a third party app, that call
> would
> never be there. I guess an mpi wrapper to deal with that could be made for
> the app, but I fell the wrapper itself, at the end, would face the same
> problem
> we face right now.
>
> My application is a genetic algorithm code that search optimal configuration
> (minimum or maximum energy) of cluster of atoms. The work flow bottleneck
> is the calculation of the cluster energy. For the cases which an analytical
> potential is available the calculation can be made internally and the workload
> is distributed among slaves nodes from a master node. This is also done
> when an analytical potential is not available and the energy calculation must
> be done externally by a quantum chemistry code like dftb+, siesta and
> Gaussian.
> So far, we have been running these codes in serial mode. No need to say that
> we could do a lot better if they could be executed in parallel.
>
> I am not familiar with DMRAA but it seems to be the right choice to deal with
> job schedulers as it covers the ones I am interested in (pbs/torque and
> loadlever).
>
> Alex
>
> 2014-12-13 7:49 GMT-02:00 Gilles Gouaillardet <gilles.gouaillar...@gmail.com>:
> George is right about the semantic
>
> However i am surprised it returns immediatly...
> That should either work or hang imho
>
> The second point is no more mpi related, and is batch manager specific.
>
> You will likely find a submit parameter to make the command block until the
> job completes. Or you can write your own wrapper.
> Or you can retrieve the jobid and qstat periodically to get the job state.
> If an api is available, this is also an option.
>
> Cheers,
>
> Gilles
>
> George Bosilca <bosi...@icl.utk.edu> wrote:
> You have to call MPI_Comm_disconnect on both sides of the intercommunicator.
> On the spawner processes you should call it on the intercom, while on the
> spawnees you should call it on the MPI_Comm_get_parent.
>
> George.
>
>> On Dec 12, 2014, at 20:43 , Alex A. Schmidt <a...@ufsm.br> wrote:
>>
>> Gilles,
>>
>> MPI_comm_disconnect seem to work but not quite.
>> The call to it returns almost immediatly while
>> the spawn processes keep piling up in the background
>> until they are all done...
>>
>> I think system('env -i qsub...') to launch the third party apps
>> would take the execution of every call back to the scheduler
>> queue. How would I track each one for their completion?
>>
>> Alex
>>
>> 2014-12-12 22:35 GMT-02:00 Gilles Gouaillardet
>> <gilles.gouaillar...@gmail.com>:
>> Alex,
>>
>> You need MPI_Comm_disconnect at least.
>> I am not sure if this is 100% correct nor working.
>>
>> If you are using third party apps, why dont you do something like
>> system("env -i qsub ...")
>> with the right options to make qsub blocking or you manually wait for the
>> end of the job ?
>>
>> That looks like a much cleaner and simpler approach to me.
>>
>> Cheers,
>>
>> Gilles
>>
>> "Alex A. Schmidt" <a...@ufsm.br> wrote:
>> Hello Gilles,
>>
>> Ok, I believe I have a simple toy app running as I think it should:
>> 'n' parent processes running under mpi_comm_world, each one
>> spawning its own 'm' child processes (each child group work
>> together nicely, returning the expected result for an mpi_allreduce call).
>>
>> Now, as I mentioned before, the apps I want to run in the spawned
>> processes are third party mpi apps and I don't think it will be possible
>> to exchange messages with them from my app. So, I do I tell
>> when the spawned processes have finnished running? All I have to work
>> with is the intercommunicator returned from the mpi_comm_spawn call...
>>
>> Alex
>>
>>
>>
>>
>> 2014-12-12 2:42 GMT-02:00 Alex A. Schmidt <a...@ufsm.br>:
>> Gilles,
>>
>> Well, yes, I guess....
>>
>> I'll do tests with the real third party apps and let you know.
>> These are huge quantum chemistry codes (dftb+, siesta and Gaussian)
>> which greatly benefits from a parallel environment. My code is just
>> a front end to use those, but since we have a lot of data to process
>> it also benefits from a parallel environment.
>>
>> Alex
>>
>>
>> 2014-12-12 2:30 GMT-02:00 Gilles Gouaillardet
>> <gilles.gouaillar...@iferc.org>:
>> Alex,
>>
>> just to make sure ...
>> this is the behavior you expected, right ?
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On 2014/12/12 13:27, Alex A. Schmidt wrote:
>>> Gilles,
>>>
>>> Ok, very nice!
>>>
>>> When I excute
>>>
>>> do rank=1,3
>>> call MPI_Comm_spawn('hello_world','
>>> ',5,MPI_INFO_NULL,rank,MPI_COMM_WORLD,my_intercomm,MPI_ERRCODES_IGNORE,status)
>>> enddo
>>>
>>> I do get 15 instances of the 'hello_world' app running: 5 for each parent
>>> rank 1, 2 and 3.
>>>
>>> Thanks a lot, Gilles.
>>>
>>> Best regargs,
>>>
>>> Alex
>>>
>>>
>>>
>>>
>>> 2014-12-12 1:32 GMT-02:00 Gilles Gouaillardet <
>>> gilles.gouaillar...@iferc.org
>>>> :
>>>>
>>>> Alex,
>>>>
>>>> just ask MPI_Comm_spawn to start (up to) 5 tasks via the maxprocs
>>>> parameter :
>>>>
>>>> int MPI_Comm_spawn(char *command, char *argv[], int maxprocs,
>>>> MPI_Info info,
>>>> int root, MPI_Comm comm, MPI_Comm *intercomm,
>>>> int array_of_errcodes[])
>>>>
>>>> INPUT PARAMETERS
>>>> maxprocs
>>>> - maximum number of processes to start (integer, significant
>>>> only at root)
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>>
>>>> On 2014/12/12 12:23, Alex A. Schmidt wrote:
>>>>
>>>> Hello Gilles,
>>>>
>>>> Thanks for your reply. The "env -i PATH=..." stuff seems to work!!!
>>>>
>>>> call system("sh -c 'env -i PATH=/usr/lib64/openmpi/bin:/bin mpirun -n 2
>>>> hello_world' ")
>>>>
>>>> did produce the expected result with a simple openmi "hello_world" code I
>>>> wrote.
>>>>
>>>> I might be harder though with the real third party app I have in mind. And
>>>> I realize
>>>> getting passed over a job scheduler with this approach might not work at
>>>> all...
>>>>
>>>> I have looked at the MPI_Comm_spawn call but I failed to understand how it
>>>> could help here. For instance, can I use it to launch an mpi app with the
>>>> option "-n 5" ?
>>>>
>>>> Alex
>>>>
>>>> 2014-12-12 0:36 GMT-02:00 Gilles Gouaillardet <
>>>> gilles.gouaillar...@iferc.org
>>>>
>>>>
>>>> :
>>>>
>>>> Alex,
>>>>
>>>> can you try something like
>>>> call system(sh -c 'env -i /.../mpirun -np 2 /.../app_name')
>>>>
>>>> -i start with an empty environment
>>>> that being said, you might need to set a few environment variables
>>>> manually :
>>>> env -i PATH=/bin ...
>>>>
>>>> and that being also said, this "trick" could be just a bad idea :
>>>> you might be using a scheduler, and if you empty the environment, the
>>>> scheduler
>>>> will not be aware of the "inside" run.
>>>>
>>>> on top of that, invoking system might fail depending on the interconnect
>>>> you use.
>>>>
>>>> Bottom line, i believe Ralph's reply is still valid, even if five years
>>>> have passed :
>>>> changing your workflow, or using MPI_Comm_spawn is a much better approach.
>>>>
>>>> Cheers,
>>>>
>>>> Gilles
>>>>
>>>> On 2014/12/12 11:22, Alex A. Schmidt wrote:
>>>>
>>>> Dear OpenMPI users,
>>>>
>>>> Regarding to this previous post
>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>> <http://www.open-mpi.org/community/lists/users/2009/06/9560.php>
>>>> from 2009,
>>>> I wonder if the reply
>>>> from Ralph Castain is still valid. My need is similar but quite simpler:
>>>> to make a system call from an openmpi fortran application to run a
>>>> third party openmpi application. I don't need to exchange mpi messages
>>>> with the application. I just need to read the resulting output file
>>>> generated
>>>> by it. I have tried to do the following system call from my fortran openmpi
>>>> code:
>>>>
>>>> call system("sh -c 'mpirun -n 2 app_name")
>>>>
>>>> but I get
>>>>
>>>> **********************************************************
>>>>
>>>> Open MPI does not support recursive calls of mpirun
>>>>
>>>> **********************************************************
>>>>
>>>> Is there a way to make this work?
>>>>
>>>> Best regards,
>>>>
>>>> Alex
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing
>>>> listus...@open-mpi.org
>>>>
>>>>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2014/12/25966.php
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing
>>>> listus...@open-mpi.org
>>>>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2014/12/25967.php
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing
>>>> listus...@open-mpi.org
>>>>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>>
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2014/12/25968.php
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>>
>>>> us...@open-mpi.org
>>>>
>>>> Subscription:
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> Link to this post:
>>>>
>>>> http://www.open-mpi.org/community/lists/users/2014/12/25969.php
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>>
>>> us...@open-mpi.org
>>>
>>> Subscription:
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2014/12/25970.php
>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/12/25971.php
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/12/25974.php
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2014/12/25975.php
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/25978.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/25979.php
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/25981.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/25982.php
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/25991.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26002.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26003.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26010.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26011.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26013.php
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26014.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26021.php
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/12/26026.php
