Thanks very much for your reply — that part of the documentation was very unclear, unfortunately. No GRES devices were ever mentioned in the setup, but it seemed to imply that you’d need to know all of the devices that are going to be used. I’ve now removed the GRES devices (I’m curious — what about Phi/MIC? Same thing, I guess?).
I have an otherwise-working system using GRES devices (checking out GPUs, etc.) — the only change was to enable cgroups. All of my node definitions are OK: [root@fen2 novosirj]# scontrol show node gpu002 NodeName=gpu002 Arch=x86_64 CoresPerSocket=6 CPUAlloc=1 CPUErr=0 CPUTot=12 CPULoad=0.43 Features=sandybrige Gres=gpu:8 …and… [root@gpu002 ~]# more /etc/slurm/gres.conf # 8 x Tesla M2070 NodeName=gpu[002] Name=gpu File=/dev/nvidia[0-3] CPUs=0-5 NodeName=gpu[002] Name=gpu File=/dev/nvidia[4-7] CPUs=6-11 I’m still seeing the same hang after removing /dev/nvidia*. A couple of things though — there /was/ something in slurmd.log that I think may be relevant (somehow thought it was unassociated with jobs before): [2016-08-11T14:09:42.524] launch task 5377692.0 request from [email protected] (port 10985) [2016-08-11T14:09:42.525] error: gres/mic unable to set OFFLOAD_DEVICES, no device files configured [2016-08-11T14:09:42.525] _run_prolog: run job script took usec=4 [2016-08-11T14:09:42.525] _run_prolog: prolog with lock for job 5377692 ran for 0 seconds [2016-08-11T14:09:42.530] [5377692.0] gres/mic unable to set OFFLOAD_DEVICES, no device files configured [2016-08-11T14:09:42.531] [5377692.0] gpu 0 is device number 0 [2016-08-11T14:09:42.531] [5377692.0] gpu 1 is device number 1 [2016-08-11T14:09:42.531] [5377692.0] gpu 2 is device number 2 [2016-08-11T14:09:42.531] [5377692.0] gpu 3 is device number 3 [2016-08-11T14:09:42.531] [5377692.0] gpu 4 is device number 4 [2016-08-11T14:09:42.531] [5377692.0] gpu 5 is device number 5 [2016-08-11T14:09:42.531] [5377692.0] gpu 6 is device number 6 [2016-08-11T14:09:42.531] [5377692.0] gpu 7 is device number 7 [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: /slurm/uid_109366/job_5377692: alloc=64391MB mem.limit=63875MB memsw.limit=unlimited [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: /slurm/uid_109366/job_5377692/step_0: alloc=64391MB mem.limit=63875MB memsw.limit=unlimited [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: manage devices jor job '5377692' [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:3 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:9 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:5 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:8 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:7 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:6 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:5 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:2 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:1 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:0 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:11 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:11 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:10 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:10 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:9 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:9 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:8 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:8 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:7 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:7 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:6 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:6 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:5 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:5 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:4 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:4 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:3 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:3 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:2 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:2 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:1 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:1 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 203:0 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 202:0 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 136:0 rwm [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 5:2 rwm [2016-08-11T14:09:42.534] [5377692.0] Allowing access to device 819275 rwm [2016-08-11T14:09:42.534] [5377692.0] Not allowing access to device 819275 rwm819275 rwm I’ll try adding the Gres debugging, but is there some way to figure out what this alleged device “819275” is (this number will change with each job). > On Aug 11, 2016, at 1:06 PM, Kilian Cavalotti > <[email protected]> wrote: > > > Hi Ryan, > > You probably shouldn't have /dev/nvidia* devices listed in > cgroup_allowed_devices_file.conf, Slurm will automatically manage them > and add them to the list of authorized devices in a cgroup when a job > starts. For reference, our cgroup_allowed_devices_file.conf contains > this: > > /dev/null > /dev/urandom > /dev/zero > /dev/sda* > /dev/cpu/*/* > /dev/pts/* > /dev/tty* > > Also, make sure you have the correct Gres definitions for your nodes > in slurm.conf: > > NodeName=foobar Gres=gpu:4 ... > > and that you have a matching gres.conf, with: > > NodeName=foobar Name=gpu File=/dev/nvidia[0-3] > > > Now if that still doesn't work, I'd recommend adding DebugFlags=Gres > to your slurm.conf and look at what the Slurm controller logs. > > Cheers, > -- > Kilian > -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - [email protected] || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'
