On Mar 15, 2012, at 8:46 AM, Reuti wrote: > Am 15.03.2012 um 15:37 schrieb Ralph Castain: > >> Just to be clear: I take it that the first entry is the host name, and the >> second is the number of slots allocated on that host? > > This is correct. > > >> FWIW: I see the problem. Our parser was apparently written assuming every >> line was a unique host, so it doesn't even check to see if there is >> duplication. Easy fix - can shoot it to you today. > > But even with the fix the nice value will be the same for all processes > forked there. Either all have the nice value of his low priority queue or the > high priority queue.
Agreed - nothing I can do about that, though. We only do the one qrsh call, so the daemons are going to fall into a single queue, and so will all their children. In this scenario, it isn't clear to me (from this discussion) that I can control which queue gets used - can I? Should I? > > -- Reuti > > >> On Mar 15, 2012, at 6:53 AM, Reuti wrote: >> >>> Am 15.03.2012 um 05:22 schrieb Joshua Baker-LePain: >>> >>>> On Wed, 14 Mar 2012 at 5:50pm, Ralph Castain wrote >>>> >>>>> On Mar 14, 2012, at 5:44 PM, Reuti wrote: >>>> >>>>>> (I was just typing when Ralph's message came in: I can confirm this. To >>>>>> avoid it, it would mean for Open MPI to collect all lines from the >>>>>> hostfile which are on the same machine. SGE creates entries for each >>>>>> queue/host pair in the machine file). >>>>> >>>>> Hmmm…I can take a look at the allocator module and see why we aren't >>>>> doing it. Would the host names be the same for the two queues? >>>> >>>> I can't speak authoritatively like Reuti can, but here's what a hostfile >>>> looks like on my cluster (note that all our name resolution is done via >>>> /etc/hosts -- there's no DNS involved): >>>> >>>> iq103 8 lab.q@iq103 <NULL> >>>> iq103 1 test.q@iq103 <NULL> >>>> iq104 8 lab.q@iq104 <NULL> >>>> iq104 1 test.q@iq104 <NULL> >>>> opt221 2 lab.q@opt221 <NULL> >>>> opt221 1 test.q@opt221 <NULL> >>> >>> Yes, exactly this needs to be parsed and adding up all entries therein for >>> one and the same machine. >>> >>> If you need it instantly, it could be put in a wrapper for start_proc_args >>> of the PE (and Open MPI compiled without SGE support), so that a custom >>> build machinefile can be used. In this case the rsh resp. ssh call also >>> needs to be caught. >>> >>> Often the opposite is desired in an SGE setup: tune it so that all slots >>> are coming from one queue only. >>> >>> But I still wonder whether it is possible to tune your setup in a similar >>> way: allow one slot more in the high priority queue (long,.q) in case it's >>> a parallel job, with an RQS (assuming 8 cores with one core >>> oversubscription): >>> >>> limit queues long.q pes * to slots=9 >>> limit queues long.q to slots=8 >>> >>> while you have an additonal short.q (the low priority queue) there with one >>> slot. The overall limit is still set on an exechost level to 9. The PE is >>> then only attached to long.q. >>> >>> -- Reuti >>> >>> PS: In your example you also had the case 2 slots in the low priority >>> queue, what is the actual setup in your cluster? >>> >>> >>>>>> @Ralph: it could work if SGE would have a facility to request the >>>>>> desired queue in `qrsh -inherit ...`, because then the $TMPDIR would be >>>>>> unique for each orted again (assuming its using different ports for >>>>>> each). >>>>> >>>>> Gotcha! I suspect getting the allocator to handle this cleanly is the >>>>> better solution, though. >>>> >>>> If I can help (testing patches, e.g.), let me know. >>>> >>>> -- >>>> Joshua Baker-LePain >>>> QB3 Shared Cluster Sysadmin >>>> UCSF_______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
