Am 15.03.2012 um 05:22 schrieb Joshua Baker-LePain: > On Wed, 14 Mar 2012 at 5:50pm, Ralph Castain wrote > >> On Mar 14, 2012, at 5:44 PM, Reuti wrote: > >>> (I was just typing when Ralph's message came in: I can confirm this. To >>> avoid it, it would mean for Open MPI to collect all lines from the hostfile >>> which are on the same machine. SGE creates entries for each queue/host pair >>> in the machine file). >> >> Hmmm…I can take a look at the allocator module and see why we aren't doing >> it. Would the host names be the same for the two queues? > > I can't speak authoritatively like Reuti can, but here's what a hostfile > looks like on my cluster (note that all our name resolution is done via > /etc/hosts -- there's no DNS involved): > > iq103 8 lab.q@iq103 <NULL> > iq103 1 test.q@iq103 <NULL> > iq104 8 lab.q@iq104 <NULL> > iq104 1 test.q@iq104 <NULL> > opt221 2 lab.q@opt221 <NULL> > opt221 1 test.q@opt221 <NULL>
Yes, exactly this needs to be parsed and adding up all entries therein for one and the same machine. If you need it instantly, it could be put in a wrapper for start_proc_args of the PE (and Open MPI compiled without SGE support), so that a custom build machinefile can be used. In this case the rsh resp. ssh call also needs to be caught. Often the opposite is desired in an SGE setup: tune it so that all slots are coming from one queue only. But I still wonder whether it is possible to tune your setup in a similar way: allow one slot more in the high priority queue (long,.q) in case it's a parallel job, with an RQS (assuming 8 cores with one core oversubscription): limit queues long.q pes * to slots=9 limit queues long.q to slots=8 while you have an additonal short.q (the low priority queue) there with one slot. The overall limit is still set on an exechost level to 9. The PE is then only attached to long.q. -- Reuti PS: In your example you also had the case 2 slots in the low priority queue, what is the actual setup in your cluster? >>> @Ralph: it could work if SGE would have a facility to request the desired >>> queue in `qrsh -inherit ...`, because then the $TMPDIR would be unique for >>> each orted again (assuming its using different ports for each). >> >> Gotcha! I suspect getting the allocator to handle this cleanly is the better >> solution, though. > > If I can help (testing patches, e.g.), let me know. > > -- > Joshua Baker-LePain > QB3 Shared Cluster Sysadmin > UCSF_______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
