On Wed, 2017-02-08 at 06:53:48 -0800, Steffen Grunewald wrote: > > Hi, > > after an all_nodes reservation for maintenance, a couple of jobs didn't start. > Instead, they complain about BadConstraints. > Since they are clones of jobs that ran perfectly before this is puzzling. > > Looking deeper into the job details, I believe I have found what causes this > - but the deeper reason is still unclear: > > # scontrol show job 12345 > JobState=PENDING Reason=BadConstraints Dependency=(null) > Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 > RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A > NumNodes=2 NumCPUs=32 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > TRES=cpu=32,node=1 > MinCPUsNode=32 MinMemoryNode=0 MinTmpDiskNode=0 > > The user had requested 2 (NumNodes) nodes, with a total of 32 (NumCPUs) cores. > This is OK, since the nodes have 16 cores each. > The TRES prat looks strange though as the total cpu count is still correct, > but > the number of nodes has been set to 1 only. > As a consequence, a matching node must have 32 (MinCPUsNode) cores, which is > impossible to fulfill. > > Attempts to change the values failed as > # scontrol update job=12345 MinCPUsNode=16 > returns without having changed anything, and TRES cannot be modified. > > Is there a way to adjust the values to make the jobs runnable again?
In the end, I tried to do the not-so-obvious and "changed" NumNodes and NumCPUs to the values "scontrol show job 12345" reported. The jobs went into JobHeldAdmin state again, but could be "scontrol release"d this time. Phew. (Apparently "changing" these values made Slurm recalculate the requirements, which now fit.) The question remains: > What may have caused Slurm (which had not been stopped during the reservation) > to mangle these values? And why is there an obvious discrepancy between NumNodes and NumCPUs on one hand and TRES nodes and MinCPUsNode on the other? Cheers, S
