You can also limit the number of jobs per user the backfill scheduler is
considering. All SchedulerParameters are worth a read if you haven't yet.
from the slurm.conf man page...
bf_max_job_user=#
The maximum number of jobs per user to attempt
starting with the backfill scheduler for ALL partitions. One can set
this limit to prevent users from flooding the backfill queue with jobs
that cannot start and that prevent jobs from other users to start. This
is similar to the MAXIJOB limit in Maui. The default value is 0, which
means no limit. This option applies only to
SchedulerType=sched/backfill. Also see the bf_max_job_part,
bf_max_job_test and bf_max_job_user_part=# options. Set
bf_max_job_test to a value much higher than bf_max_job_user.
bf_max_job_user_part=#
The maximum number of jobs per user per
partition to attempt starting with the backfill scheduler for any single
partition. The default value is 0, which means no limit. This option
applies only to SchedulerType=sched/backfill. Also see the
bf_max_job_part, bf_max_job_test and bf_max_job_user=# options.
On 3/18/20 1:21 PM, Sean Maxwell wrote:
Hi Mike,
I think you want to set MaxSubmitJobs on the users account
association. The parameter is described in the sacctmgr documentation
as being the maximum number of jobs a user can have in state running
or pending.
https://slurm.schedmd.com/sacctmgr.html
Thanks,
-Sean
On Wed, Mar 18, 2020 at 1:12 PM Hanby, Mike <mha...@uab.edu
<mailto:mha...@uab.edu>> wrote:
Howdy,
We are running Slurm 18.08. We have a user who has, twice,
submitted over 15 thousand jobs to the cluster (the queue normally
has a couple thousand jobs at any given time).
This results in Slurm being unresponsive to user requests / job
submits. I suspect the scheduler is getting bogged down doing
backfill processing.
Is there any way to limit the maximum number of jobs a single user
can have in the queue at any given time?
----------------
Mike Hanby
mhanby @ uab.edu <http://uab.edu>
Systems Analyst III - Enterprise
IT Research Computing Services
The University of Alabama at Birmingham
