Gilles, Yep, that is what I am doing. I have the child processes wait for a message from the master before doing anything. Now I have to find a creative way to determine which processes to scale back as I allocate the work. I was just using modulo on the line number from the input, but if my divisor changes, it can cause race conditions I don't want.
Thanks for the feedback from both of you. Brian Andrus From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gilles Gouaillardet Sent: Thursday, March 17, 2016 4:59 PM To: Open MPI Users Subject: Re: [OMPI users] Dynamically throttle/scale processes Brian, unlike Ralph, i will assume all your processes are MPI tasks. at first glance, the MPI philosophy is the other way around : start with mpirun -np 1 traffic_cop, and then MPI_Comm_spawn("child") when you need more workers. that being said, if you are fine with having idle children (e.g. children that consume no CPU resources, but do keep memory, network and other system resources allocated), then you can start 256 mpi tasks, either with mpirun -np 256 cop_children or mpirun -np 1 traffic_cop : -np 255 children /*i am not 100% sure about the syntax here ... */ there is no MPI way to signal a task, but you can have your children wait for a message from the master. unless you are using a TCP interconnect, i do no think OpenMPI is production ready for MPI_THREAD_MULTIPLE, so one option is to have your children MPI_Recv() information from the traffic cop in the main process, and do the real job is an other pthread (so the main process can kill the working thread when MPI_Recv returns) an other option is to MPI_Irecv(), do the job and periodically MPI_Test() to check if there is any order from the traffic cop. Cheers, Gilles On 3/18/2016 8:38 AM, Ralph Castain wrote: Hmmm....I haven't heard of that specific use-case, but I have seen some similar things. Did you want the processes to be paused, or killed, when you scale down? Obviously, I'm assuming they are not MPI procs, yes? I can certainly see a way to make mpirun do it without too much fuss, though it would require a message as opposed to a signal so you can indicate how many procs to "idle/kill". On Mar 17, 2016, at 3:22 PM, Andrus, Brian Contractor <bdand...@nps.edu<mailto:bdand...@nps.edu>> wrote: All, I have an mpi-based program that has a master process that acts as a 'traffic cop' in that it hands out work to child processes. I want to be able to dynamically throttle how many child processes are in use at any given time. For instance, if I start it with "mpirun -n 512" I could send a signal to tell it to only use 256 of them for a bit and then tell it to scale back up. The upper limit being the number of processes mpirun was launched with. Has anyone done anything like this? Maybe a better way to do it? Basically my program is crunching through a large text file, examining each line for various things. Thanks in advance for any insight, Brian Andrus _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/03/28744.php _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/03/28745.php
