Found the problem: another job was blocking access to the reservation.
The strangest thing is that the node (gpu03) has always been reserved
for a project, the blocking job did not explicitly request it (and even
if it did, it would have been denied access) but its state was:
JobState=PENDING Reason=ReqNodeNotAvail,_UnavailableNodes:gpu03
Dependency=(null)
Paint me surprised...
Diego
Il 07/12/2024 10:03, Diego Zuccato via slurm-users ha scritto:
Ciao Davide.
Il 06/12/2024 16:42, Davide DelVento ha scritto:
I find it extremely hard to understand situations like this. I wish
Slurm were more clear on how it reported what it is doing, but I
digress...
I agree. A "scontrol explain" command could be really useful to pinpont
the cause :)
I suspect that there are other job(s) which have higher priority than
this one which are supposed to run on that node but cannot start
because maybe this/these high-priority jobs(s) need(s) several nodes
and the other nodes are not available at the moment?
That partition is a single node, and it's IDLE. If another job needed
it, it would be in PLANNED state (IIRC).
Pure speculation, obviously, since I have no idea what the rest of
your cluster looks like, and what the rest of the workflow is, but the
clue/ hint is
> JobState=PENDING Reason=Priority Dependency=(null)
You are pending because something else has higher priority. Going back
to my first sentence, I wish Slurm would say which one other job
(maybe there are more than one, but one would suffice for this
investigation) is trumping this job priority so one could more
clearly understand what is going on, without sleuthing.
Couldn't agree more :) Scheduler is quite opaque in its decisions. :(
Actually the job that the user submitted is not starting and has
Reason=PartitionConfig . But QoS 'debug' (the one I'm using for testing)
does have higher priority (1000) than QoS 'long' (10, IIRC).
Diego
On Fri, Dec 6, 2024 at 7:36 AM Diego Zuccato via slurm-users <slurm-
us...@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com>> wrote:
Hello all.
An user reported that a job wasn't starting, so I tried to replicate
the
request and I get:
-8<--
[root@ophfe1 root.old]# scontrol show job 113936
JobId=113936 JobName=test.sh
UserId=root(0) GroupId=root(0) MCS_label=N/A
Priority=1 Nice=0 Account=root QOS=long
JobState=PENDING Reason=Priority Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A
SubmitTime=2024-12-06T13:19:36 EligibleTime=2024-12-06T13:19:36
AccrueTime=2024-12-06T13:19:36
StartTime=Unknown EndTime=Unknown Deadline=N/A
SuspendTime=None SecsPreSuspend=0
LastSchedEval=2024-12-06T13:21:32
Scheduler=Backfill:*
Partition=m3 AllocNode:Sid=ophfe1:855189
ReqNodeList=(null) ExcNodeList=(null)
NodeList=
NumNodes=1-1 NumCPUs=96 NumTasks=96 CPUs/Task=1
ReqB:S:C:T=0:0:*:*
TRES=cpu=96,mem=95000M,node=1,billing=1296
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=95000M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/home/root.old/test.sh
WorkDir=/home/root.old
StdErr=/home/root.old/%N-%J.err
StdIn=/dev/null
StdOut=/home/root.old/%N-%J.out
Power=
[root@ophfe1 root.old]# scontrol sho partition m3
PartitionName=m3
AllowGroups=ALL DenyAccounts=formazione AllowQos=ALL
AllocNodes=ALL Default=NO QoS=N/A
DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
Hidden=NO
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
MaxCPUsPerNode=UNLIMITED
Nodes=mtx20
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
OverSubscribe=NO
OverTimeLimit=NONE PreemptMode=CANCEL
State=UP TotalCPUs=192 TotalNodes=1
SelectTypeParameters=CR_SOCKET_MEMORY
JobDefaults=(null)
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=192,mem=1150000M,node=1,billing=2592
TRESBillingWeights=CPU=13.500,Mem=2.2378G
[root@ophfe1 root.old]# scontrol show node mtx20
NodeName=mtx20 Arch=x86_64 CoresPerSocket=24
CPUAlloc=0 CPUEfctv=192 CPUTot=192 CPULoad=0.00
AvailableFeatures=ib,matrix,intel,avx
ActiveFeatures=ib,matrix,intel,avx
Gres=(null)
NodeAddr=mtx20 NodeHostName=mtx20 Version=22.05.6
OS=Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47
UTC 2022
RealMemory=1150000 AllocMem=0 FreeMem=1156606 Sockets=4 Boards=1
MemSpecLimit=2048
State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=8 Owner=N/A
MCS_label=N/A
Partitions=m3
BootTime=2024-12-06T10:01:42 SlurmdStartTime=2024-12-06T10:02:54
LastBusyTime=2024-12-06T10:51:58
CfgTRES=cpu=192,mem=1150000M,billing=2592
AllocTRES=
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
-8<--
So the node is free, the partition does not impose extra limits (used
only for accounting factors) but the job does not start.
Any hints?
Tks
-- Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
-- slurm-users mailing list -- slurm-users@lists.schedmd.com
<mailto:slurm-users@lists.schedmd.com>
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
<mailto:slurm-users-le...@lists.schedmd.com>
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
