Following on from what Michael said, the default Slurm configuration is to allocate only one job per node. If GRES a100_1g.10gb is on the same node ensure to enable “SelectType=select/cons_res” (info at https://slurm.schedmd.com/cons_res.html) to permit multiple jobs to use the same node.
Also using “TaskPlugin=task/cgroup” is useful to ensure that users cannot inadvertently access resources not allocated to other jobs on the same node (refer to the slurm.conf man page). -Greg From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Michael Gutteridge <michael.gutteri...@gmail.com> Date: Wednesday, 2 August 2023 at 5:22 pm To: Slurm User Community List <slurm-users@lists.schedmd.com> Subject: [EXTERNAL] Re: [slurm-users] Job in "priority" status - resources available I'm not sure there's enough information in your message- Slurm version and configs are often necessary to make a more confident diagnosis. However, the behaviour you are looking for (lower priority jobs skipping the line) is called "backfill". There's docs here: https://slurm.schedmd.com/sched_config.html#backfill<https://urldefense.com/v3/__https:/slurm.schedmd.com/sched_config.html*backfill__;Iw!!Nmw4Hv0!yN2vkwBRGKx9XuVO3o7g6Ca8yN0A5bXazd8I1g0g1FaKEOi2P0xAg6Z_1eyqdwOFsvIv0D64pDlw4G36wojaY0uNkOhsh72u8g$> It should be loaded and active by default which is why I'm not super confident here. There may also be something else going on with the node configuration as it looks like 1596 would maybe need the same node? Maybe there's not enough CPU or memory to accommodate both jobs (1596 and 1739)? HTH - Michael On Wed, Aug 2, 2023 at 5:13 AM Cumer Cristiano <cristianomaria.cu...@unibz.it<mailto:cristianomaria.cu...@unibz.it>> wrote: Hello, I'm quite a newbie regarding Slurm. I recently created a small Slurm instance to manage our GPU resources. I have this situation: JOBID STATE TIME ACCOUNT PARTITION PRIORITY REASON CPU MIN_MEM TRES_PER_NODE 1739 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100_1g.10gb:1 1738 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100-sxm4-80gb:1 1737 PENDING 0:00 standard gpu-low 5 Priority 1 80G gres:gpu:a100-sxm4-80gb:1 1736 PENDING 0:00 standard gpu-low 5 Resources 1 80G gres:gpu:a100-sxm4-80gb:1 1740 PENDING 0:00 standard gpu-low 1 Priority 1 8G gres:gpu:a100_3g.39gb 1735 PENDING 0:00 standard gpu-low 1 Priority 8 64G gres:gpu:a100-sxm4-80gb:1 1596 RUNNING 1-13:26:45 standard gpu-low 3 None 2 64G gres:gpu:a100_1g.10gb:1 1653 RUNNING 21:09:52 standard gpu-low 2 None 1 16G gres:gpu:1 1734 RUNNING 59:52 standard gpu-low 1 None 8 64G gres:gpu:a100-sxm4-80gb:1 1733 RUNNING 1:01:54 standard gpu-low 1 None 8 64G gres:gpu:a100-sxm4-80gb:1 1732 RUNNING 1:02:39 standard gpu-low 1 None 8 40G gres:gpu:a100-sxm4-80gb:1 1731 RUNNING 1:08:28 standard gpu-low 1 None 8 40G gres:gpu:a100-sxm4-80gb:1 1718 RUNNING 10:16:40 standard gpu-low 1 None 2 8G gres:gpu:v100 1630 RUNNING 1-00:21:21 standard gpu-low 1 None 1 30G gres:gpu:a100_3g.39gb 1610 RUNNING 1-09:53:23 standard gpu-low 1 None 2 8G gres:gpu:v100 Job 1736 is in the PENDING state since there are no more available a100-sxm4-80gb GPUs. The job priority starts to rise with time (priority 5) as expected. Now another user submits job 1739 on a gres:gpu:a100_1g.10gb:1 that is available, but the job is not starting since its priority is 1. This is obviously not the desired outcome, and I believe I must change the scheduling strategy. Could someone with more experience than me give me some hints? Thanks, Cristiano
