The DEPTH_OBLIVIOUS part may not be what you want, but the SMALL_RELATIVE_TO_TIME may help. The DEPTH_OBVLIVIOUS is because we have a very deep hierarchy used for accounting and are on 14.03 which doesn't have FAIR_TREE.
One way I wrapped my head around PriorityFavorSmall=NO and SMALL_RELATIVE_TO_TIME was submit test jobs with a --begin value long enough for them to stay pending a few minutes. I then evaluated the priorities of each job using "sprio" command to see how various changes impacted the job size priority. Something like this: # Jobs of equal size but different times sbatch -p background --begin=now+120 --time=01:00:00 --nodes=1 --ntasks-per-node=8 sleep.slrm sbatch -p background --begin=now+120 --time=02:00:00 --nodes=1 --ntasks-per-node=8 sleep.slrm # Jobs of equal time but different sizes sbatch -p background --begin=now+120 --time=02:00:00 --nodes=1 --ntasks-per-node=8 sleep.slrm sbatch -p background --begin=now+120 --time=02:00:00 --nodes=1 --ntasks-per-node=1 sleep.slrm Another technique we have used is setting varied UsageFactor values on the different QOS. So the usage factor of something like background, that can be preempted, is 0.25 while the usage factor for non-preemptable partitions is 1.0. This means 4 CPU hours in background only count as 1 CPU hour in terms of usage accounting against fairshare. FairShare is difficult to get right at first, takes some trial and error. We've found no matter how well we test various scenarios, our users will always find a way to come up with new usage patterns that require us to re-evaluate our configuration. - Trey ============================= Trey Dockendorf Systems Analyst I Texas A&M University Academy for Advanced Telecommunications and Learning Technologies Phone: (979)458-2396 Email: [email protected] Jabber: [email protected] On Sun, Aug 30, 2015 at 11:38 AM, Gerry Creager - NOAA Affiliate < [email protected]> wrote: > Trey > > This helps a lot. Thank you! I'll be reconfiguring > PriorityFavorSmall=NO > PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME > > shortly, and see about changing MAXJOBS and MAXSUBMIT some more to tweak > how he's handled. > > Gerry > > On Sun, Aug 30, 2015 at 4:11 PM, Trey Dockendorf <[email protected]> > wrote: > >> You may have to adjust how you prioritize jobs by size and wallclock time. >> >> PriorityFavorSmall=NO >> PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME >> >> The PriorityFavorSmall and SMALL_RELATIVE_TO_TIME is used by us to favor >> both short and large jobs. So if two jobs are equal in size, the shorter >> of the two is favored. Also if two jobs are equal in time, the larger is >> favored. We use this as a way to get short jobs in and out of the queues >> quickly as well as help large jobs (typically MPI) have priority over small >> serial jobs. >> >> How much weight the job size is given for priority will be determined >> also by PriorityWeightJobSize. >> >> A small job getting in before big jobs also has to do with how backfill >> is configured. If the small job starting won't impact a higher priority >> large job's start time, it should get backfilled. >> >> You can use 'sshare' to view the fair share values for users. You can >> use 'sprio' to inspect a pending job's various priority factors. So if you >> inspect this particular user's jobs against that of someone who had to >> wait, you may see why SLURM favored those jobs. One thing I did when >> planning our FairShare was create a handful of temporary accounts to >> represent our various users and then submit batches of jobs that just >> executed 'sleep' to see how various scenarios would effect their fairshare >> and priorities. >> >> Using MaxJobs and MaxSubmit in the accounting database is one way to >> limit overly aggressive usage. We've done such limitations via a QOS, and >> then we set that QOS as the user's default. We also use the preemptable >> partitions to allow for aggressive use, at the cost of potentially being >> preempted. The partitions that can't be preempted are all restricted on >> how much they can monopolize the cluster. The QOS assigned to each >> partition is automatically assigned to all jobs in that partition via >> job_submit.lua plugin. We enforce some QOS usage this way so that >> problematic users are always forced to use a specific QOS, and that QOS >> value comes from their DefaultQOS set on their user record in accounting >> database. >> >> In theory, if your FairShare is configured to account for this type of >> aggressive usage, the person running those jobs would begin to have a lower >> and lower priority. Then at some point those other users who had to wait >> would begin to have a higher priority in relation to the aggressive user >> and would no longer have to wait. >> >> Currently we use QOS to set limits but also split up our resources by >> stakeholder partitions , non-stakeholder partitions and then the >> "background" partitions (that can be preempted). Should testing of SUSPEND >> preemption via QOS go well in 15.08 RC, we plan to move away from >> stakeholder partitions and handle everything via QOS and FairShare >> (FAIR_TREE). >> >> - Trey >> >> ============================= >> >> Trey Dockendorf >> Systems Analyst I >> Texas A&M University >> Academy for Advanced Telecommunications and Learning Technologies >> Phone: (979)458-2396 >> Email: [email protected] >> Jabber: [email protected] >> >> On Thu, Aug 27, 2015 at 4:55 PM, Gerry Creager - NOAA Affiliate < >> [email protected]> wrote: >> >>> I'm running slurm on a Cray XE6m, and a relatively small one at that... >>> with fair-share enabled. Today we had an episode where a single user was >>> effectively snagging the entire machine with small, short (wallclock) jobs >>> while all other users had to wait. >>> >>> He's using array control (--array=1-21), requesting 4 nodes, and >>> generally a 1-hr wall clock limit. And occupying a significant portion of >>> the machine. >>> >>> I've already restricted him (sacctmgr) to maxjobs=16, maxsubmit=32. If I >>> understand the parameters correctly, that should reduce the number of jobs, >>> and thus, nodes, he's consuming at a given time. >>> >>> I'm considering telling him to restrict the number of simultaneous array >>> members to n=4, and reducing his maxnodes to 1, which will result in a >>> slower run, but should still be managable. >>> >>> Are there other options/parameters I'm missing here, in configuring >>> slurm for fair-share? It doesn't seem like it was a very fair distribution >>> of resources... >>> >>> Thanks >>> gerry >>> -- >>> Gerry Creager >>> NSSL/CIMMS >>> 405.325.6371 >>> ++++++++++++++++++++++ >>> “Big whorls have little whorls, >>> That feed on their velocity; >>> And little whorls have lesser whorls, >>> And so on to viscosity.” >>> Lewis Fry Richardson (1881-1953) >>> >> >> > > > -- > Gerry Creager > NSSL/CIMMS > 405.325.6371 > ++++++++++++++++++++++ > “Big whorls have little whorls, > That feed on their velocity; > And little whorls have lesser whorls, > And so on to viscosity.” > Lewis Fry Richardson (1881-1953) >
