Hello, I can't understand why jobs---even without asking for GPU resources---don't get scheduled. I must have something fundamentally wrong in the configuration. Maybe someone can help.
I have 2 machines (physically different apparatuses---I suppose in SLURM parlance this is a node, but I am not sure about that), with resp. 1 and 2 GPUs, and each with (I believe) 6 hyperthreaded CPUs. I would like to be able to schedule either normal CPU jobs (gres=gpu:0) at a granularity 1 job / CPU (so that I can run 12 jobs in parallel), or GPU jobs (gres=gpu:1) at a granularity 1 job / GPU, requiring additionally 1 CPU, so that I can run 3 GPU jobs in parallel. In that case, there should be still room for 9 single-threaded jobs on the cluster (well, maybe not a cluster, but rather a binary system). But in trying to tell SLURM about the gpu's, it has stopped completely scheduling jobs. Even jobs where I don't even want a GPU. Slurm claims that "gres/gpu count too low (0 < 1)"---but I have to clue as to what the 0 and the 1 refer to (claimed/detected/physical/reserved/available/required gpus?). # grep gpu /etc/slurm-llnl/slurm.conf GresTypes=gpu NodeName=deep-novo-1 RealMemory=32145 CPUS=12 Sockets=1 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:1 NodeName=deep-novo-2 RealMemory=129105 CPUS=12 Sockets=1 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:2 # grep gpu /etc/slurm-llnl/gres.conf NodeName=deep-novo-1 Name=gpu File=/dev/nvidia0 NodeName=deep-novo-2 Name=gpu File=/dev/nvidia[0-1] deep-novo-1# grep gpu /var/log/slurm-llnl/slurmd.log [2016-12-06T10:40:13.856] Gres Name=gpu Count=1 ID=7696487 File=/dev/nvidia0 [2016-12-06T10:40:13.856] gpu 0 is device number 0 # scontrol show nodes NodeName=deep-novo-1 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=12 CPULoad=0.02 Features=(null) Gres=gpu:1 NodeAddr=deep-novo-1 NodeHostName=deep-novo-1 Version=14.03 OS=Linux RealMemory=32145 AllocMem=0 Sockets=1 Boards=1 State=IDLE+DRAIN ThreadsPerCore=2 TmpDisk=0 Weight=1 BootTime=2016-12-01T15:02:22 SlurmdStartTime=2016-12-06T10:40:13 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Reason=gres/gpu count too low (0 < 1) [root@2016-12-02T15:16:44] NodeName=deep-novo-2 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=12 CPULoad=0.01 Features=(null) Gres=gpu:2 NodeAddr=deep-novo-2 NodeHostName=deep-novo-2 Version=14.03 OS=Linux RealMemory=129105 AllocMem=0 Sockets=1 Boards=1 State=IDLE+DRAIN ThreadsPerCore=2 TmpDisk=0 Weight=1 BootTime=2016-11-28T14:34:59 SlurmdStartTime=2016-12-06T10:40:16 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Reason=gres/gpu count too low (0 < 2) [root@2016-12-02T15:16:44] Any help would be greatly appreciated. In particular, apointers to a more verbose reasoning about why a job can / can't be scheduled would be fantastic. ---david -- David van Leeuwen
