On 08/24/2017 07:02 PM, Patrick Goetz wrote:
I'm managing what has turned into a very busy cluster and the users are complaining about resource allocation. There appear to be 2 main complaints: 1. When users submit (say) 8 long running single core jobs, it doesn't appear that Slurm attempts to consolidate them on a single node (each of our nodes can accommodate 16 tasks). This becomes problematic for users submitting threaded applications which must run on a single node, as even short jobs can be locked out of every node on the system for days at a time. 2. Users are interested in temporally influenced resource allocation. Quoting one of the users "heavy users in recent past will get fewer resources assigned to them, while users that rarely use the cluster will likely get more jobs run. We believe a good system will track each user base on their usage" Basically they want to keep track of User_usage = Sum (Number of cores requested x Running time) accumulated over, say, a month's time before the counter is reset. I'm not really sure how to implement such a thing, or if it is even possible. It's quite possible that a solution to #1 would effectively solve #2 for them, as #1 is the source of the request for #2.
Hi Patrick, Just my two pennies' worth of ideas below. COMPLAINT 1 You might put some of the nodes in a special partition, where a user needs to allocate the full node to be able to use it. Or you might say that users can allocate full nodes only, and give them the responsibility to make the best use of the node by starting several parallel computations on them in their job scripts. COMPLAINT 2 We have a tool "projinfo" that tells how many core hours projects that have been used during the last 30 days. It uses accounting information logged in files (slurm.conf configuration lines "JobCompLoc=/etc/slurm/slurm_jobcomp_logger" and "JobCompType=jobcomp/script"). Another way to implement such a tool might use job information in the Slurm database. We also give "grants" to projects, saying how many core hours a project may use during the last 30 days. When the grant is overdrafted, job priority in the Slurm queue goes down We do not use FairShare, but instead a home-made scripted scheduling, that is more like FIFO (First In, First Out), with some additions that makes it difficult for the early bird to monopolize the cluster. Here is some information to our users about it: http://uppmax.uu.se/support/faq/running-jobs-faq/your-priority-in-the-waiting-job-queue/ Cheers, -- Lennart Karlsson, UPPMAX, Uppsala University, Sweden http://uppmax.uu.se/
