I’m also noticing that I sometimes see this after a few minutes: [novosirj@fen2 ~]$ srun -n1 -p tesla --gres=gpu:1 --reservation=cgroups_test --pty --time=20:00 bash -i srun: error: pty: accept failure: Interrupted system call
But things work fine without the --gres definition. I’m including an excerpt of the DebugFlags=Gres output here. Much of this is repeated every minute or so while the job is running: [2016-08-11T14:27:42.586] gres: gpu state for job 5377696 [2016-08-11T14:27:42.586] gres_cnt:1 node_cnt:0 type:(null) [2016-08-11T14:27:42.586] gres/gpu: state for gpu002 [2016-08-11T14:27:42.586] gres_cnt found:8 configured:8 avail:8 alloc:1 [2016-08-11T14:27:42.586] gres_bit_alloc:0 [2016-08-11T14:27:42.586] gres_used:(null) [2016-08-11T14:27:42.586] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:42.586] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:42.586] topo_gres_cnt_alloc[0]:1 [2016-08-11T14:27:42.586] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:42.586] type[0]:(null) [2016-08-11T14:27:42.586] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:42.587] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:42.587] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:42.587] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:42.587] type[1]:(null) [2016-08-11T14:27:42.587] gres/mic: state for gpu002 [2016-08-11T14:27:42.587] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:42.587] gres_bit_alloc:NULL [2016-08-11T14:27:42.587] gres_used:(null) [2016-08-11T14:27:42.587] gres: gpu state for job 5377696 [2016-08-11T14:27:42.587] gres_cnt:1 node_cnt:1 type:(null) [2016-08-11T14:27:42.588] gres_bit_step_alloc:NULL [2016-08-11T14:27:42.588] gres_bit_alloc[0]:0 [2016-08-11T14:27:42.588] gres_cnt_step_alloc[0]:0 [2016-08-11T14:27:42.588] sched: _slurm_rpc_allocate_resources JobId=5377696 NodeList=gpu002 usec=2968 [2016-08-11T14:27:42.593] gres/gpu state for step 5377696.0 [2016-08-11T14:27:42.593] gres_cnt:1 node_cnt:0 type:(null) [2016-08-11T14:27:42.593] node_in_use:NULL [2016-08-11T14:27:42.593] gres/gpu state for step 5377696.0 [2016-08-11T14:27:42.593] gres_cnt:1 node_cnt:1 type:(null) [2016-08-11T14:27:42.593] gres_bit_alloc[0]:0 [2016-08-11T14:27:48.161] gres/gpu: state for gpu002 [2016-08-11T14:27:48.161] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.161] gres_bit_alloc: [2016-08-11T14:27:48.161] gres_used:(null) [2016-08-11T14:27:48.161] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.161] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.161] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.161] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.161] type[0]:(null) [2016-08-11T14:27:48.162] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.162] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.162] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.162] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.162] type[1]:(null) [2016-08-11T14:27:48.162] gres/mic: state for gpu002 [2016-08-11T14:27:48.162] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.162] gres_bit_alloc:NULL [2016-08-11T14:27:48.162] gres_used:(null) [2016-08-11T14:27:48.162] gres/gpu: state for gpu002 [2016-08-11T14:27:48.162] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.162] gres_bit_alloc: [2016-08-11T14:27:48.162] gres_used:(null) [2016-08-11T14:27:48.162] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.162] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.162] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.162] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.162] type[0]:(null) [2016-08-11T14:27:48.162] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.162] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.162] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.162] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.163] type[1]:(null) [2016-08-11T14:27:48.163] gres/mic: state for gpu002 [2016-08-11T14:27:48.163] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.163] gres_bit_alloc:NULL [2016-08-11T14:27:48.163] gres_used:(null) [2016-08-11T14:27:48.163] gres/gpu: state for gpu002 [2016-08-11T14:27:48.163] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.163] gres_bit_alloc: [2016-08-11T14:27:48.163] gres_used:(null) [2016-08-11T14:27:48.163] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.163] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.163] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.163] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.163] type[0]:(null) [2016-08-11T14:27:48.163] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.163] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.163] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.163] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.163] type[1]:(null) [2016-08-11T14:27:48.163] gres/mic: state for gpu002 [2016-08-11T14:27:48.163] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.163] gres_bit_alloc:NULL [2016-08-11T14:27:48.163] gres_used:(null) [2016-08-11T14:27:48.164] gres/gpu: state for gpu002 [2016-08-11T14:27:48.164] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.164] gres_bit_alloc: [2016-08-11T14:27:48.164] gres_used:(null) [2016-08-11T14:27:48.164] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.164] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.164] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.164] type[0]:(null) [2016-08-11T14:27:48.164] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.164] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.164] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.164] type[1]:(null) [2016-08-11T14:27:48.164] gres/mic: state for gpu002 [2016-08-11T14:27:48.164] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.164] gres_bit_alloc:NULL [2016-08-11T14:27:48.164] gres_used:(null) [2016-08-11T14:27:48.164] gres/gpu: state for gpu002 [2016-08-11T14:27:48.164] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.164] gres_bit_alloc: [2016-08-11T14:27:48.164] gres_used:(null) [2016-08-11T14:27:48.164] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.164] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.164] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.164] type[0]:(null) [2016-08-11T14:27:48.164] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.164] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.165] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.165] type[1]:(null) [2016-08-11T14:27:48.165] gres/mic: state for gpu002 [2016-08-11T14:27:48.165] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.165] gres_bit_alloc:NULL [2016-08-11T14:27:48.165] gres_used:(null) [2016-08-11T14:27:48.165] gres/gpu: state for gpu002 [2016-08-11T14:27:48.165] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.165] gres_bit_alloc: [2016-08-11T14:27:48.165] gres_used:(null) [2016-08-11T14:27:48.165] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.165] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.165] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.165] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.165] type[0]:(null) [2016-08-11T14:27:48.165] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.165] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.165] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.165] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.165] type[1]:(null) [2016-08-11T14:27:48.165] gres/mic: state for gpu002 [2016-08-11T14:27:48.165] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.165] gres_bit_alloc:NULL [2016-08-11T14:27:48.165] gres_used:(null) [2016-08-11T14:27:48.166] gres/gpu: state for gpu002 [2016-08-11T14:27:48.166] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.166] gres_bit_alloc: [2016-08-11T14:27:48.166] gres_used:(null) [2016-08-11T14:27:48.166] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.166] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.166] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.166] type[0]:(null) [2016-08-11T14:27:48.166] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.166] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.166] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.166] type[1]:(null) [2016-08-11T14:27:48.166] gres/mic: state for gpu002 [2016-08-11T14:27:48.166] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.166] gres_bit_alloc:NULL [2016-08-11T14:27:48.166] gres_used:(null) [2016-08-11T14:27:48.166] gres/gpu: state for gpu002 [2016-08-11T14:27:48.166] gres_cnt found:8 configured:8 avail:8 alloc:0 [2016-08-11T14:27:48.166] gres_bit_alloc: [2016-08-11T14:27:48.166] gres_used:(null) [2016-08-11T14:27:48.166] topo_cpus_bitmap[0]:0-5 [2016-08-11T14:27:48.166] topo_gres_bitmap[0]:0-3 [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[0]:0 [2016-08-11T14:27:48.166] topo_gres_cnt_avail[0]:4 [2016-08-11T14:27:48.166] type[0]:(null) [2016-08-11T14:27:48.166] topo_cpus_bitmap[1]:6-11 [2016-08-11T14:27:48.166] topo_gres_bitmap[1]:4-7 [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[1]:0 [2016-08-11T14:27:48.166] topo_gres_cnt_avail[1]:4 [2016-08-11T14:27:48.166] type[1]:(null) [2016-08-11T14:27:48.166] gres/mic: state for gpu002 [2016-08-11T14:27:48.166] gres_cnt found:0 configured:0 avail:0 alloc:0 [2016-08-11T14:27:48.166] gres_bit_alloc:NULL [2016-08-11T14:27:48.166] gres_used:(null) > On Aug 11, 2016, at 2:11 PM, Ryan Novosielski <[email protected]> wrote: > > Thanks very much for your reply — that part of the documentation was very > unclear, unfortunately. No GRES devices were ever mentioned in the setup, but > it seemed to imply that you’d need to know all of the devices that are going > to be used. I’ve now removed the GRES devices (I’m curious — what about > Phi/MIC? Same thing, I guess?). > > I have an otherwise-working system using GRES devices (checking out GPUs, > etc.) — the only change was to enable cgroups. All of my node definitions are > OK: > > [root@fen2 novosirj]# scontrol show node gpu002 > NodeName=gpu002 Arch=x86_64 CoresPerSocket=6 > CPUAlloc=1 CPUErr=0 CPUTot=12 CPULoad=0.43 Features=sandybrige > Gres=gpu:8 > > …and… > > [root@gpu002 ~]# more /etc/slurm/gres.conf > # 8 x Tesla M2070 > NodeName=gpu[002] Name=gpu File=/dev/nvidia[0-3] CPUs=0-5 > NodeName=gpu[002] Name=gpu File=/dev/nvidia[4-7] CPUs=6-11 > > I’m still seeing the same hang after removing /dev/nvidia*. A couple of > things though — there /was/ something in slurmd.log that I think may be > relevant (somehow thought it was unassociated with jobs before): > > [2016-08-11T14:09:42.524] launch task 5377692.0 request from > [email protected] (port 10985) > [2016-08-11T14:09:42.525] error: gres/mic unable to set OFFLOAD_DEVICES, no > device files configured > [2016-08-11T14:09:42.525] _run_prolog: run job script took usec=4 > [2016-08-11T14:09:42.525] _run_prolog: prolog with lock for job 5377692 ran > for 0 seconds > [2016-08-11T14:09:42.530] [5377692.0] gres/mic unable to set OFFLOAD_DEVICES, > no device files configured > [2016-08-11T14:09:42.531] [5377692.0] gpu 0 is device number 0 > [2016-08-11T14:09:42.531] [5377692.0] gpu 1 is device number 1 > [2016-08-11T14:09:42.531] [5377692.0] gpu 2 is device number 2 > [2016-08-11T14:09:42.531] [5377692.0] gpu 3 is device number 3 > [2016-08-11T14:09:42.531] [5377692.0] gpu 4 is device number 4 > [2016-08-11T14:09:42.531] [5377692.0] gpu 5 is device number 5 > [2016-08-11T14:09:42.531] [5377692.0] gpu 6 is device number 6 > [2016-08-11T14:09:42.531] [5377692.0] gpu 7 is device number 7 > [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: > /slurm/uid_109366/job_5377692: alloc=64391MB mem.limit=63875MB > memsw.limit=unlimited > [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: > /slurm/uid_109366/job_5377692/step_0: alloc=64391MB mem.limit=63875MB > memsw.limit=unlimited > [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: manage devices jor job > '5377692' > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:3 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:9 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:5 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:8 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:7 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:6 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:5 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:2 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:1 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:0 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:11 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:11 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:10 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:10 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:9 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:9 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:8 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:8 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:7 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:7 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:6 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:6 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:5 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:5 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:4 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:4 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:3 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:3 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:2 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:2 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:1 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:1 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 203:0 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 202:0 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c > 136:0 rwm > [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 5:2 > rwm > [2016-08-11T14:09:42.534] [5377692.0] Allowing access to device 819275 rwm > [2016-08-11T14:09:42.534] [5377692.0] Not allowing access to device 819275 > rwm819275 rwm > > I’ll try adding the Gres debugging, but is there some way to figure out what > this alleged device “819275” is (this number will change with each job). > >> On Aug 11, 2016, at 1:06 PM, Kilian Cavalotti >> <[email protected]> wrote: >> >> >> Hi Ryan, >> >> You probably shouldn't have /dev/nvidia* devices listed in >> cgroup_allowed_devices_file.conf, Slurm will automatically manage them >> and add them to the list of authorized devices in a cgroup when a job >> starts. For reference, our cgroup_allowed_devices_file.conf contains >> this: >> >> /dev/null >> /dev/urandom >> /dev/zero >> /dev/sda* >> /dev/cpu/*/* >> /dev/pts/* >> /dev/tty* >> >> Also, make sure you have the correct Gres definitions for your nodes >> in slurm.conf: >> >> NodeName=foobar Gres=gpu:4 ... >> >> and that you have a matching gres.conf, with: >> >> NodeName=foobar Name=gpu File=/dev/nvidia[0-3] >> >> >> Now if that still doesn't work, I'd recommend adding DebugFlags=Gres >> to your slurm.conf and look at what the Slurm controller logs. >> >> Cheers, >> -- >> Kilian >> > > -- > ____ > || \\UTGERS, > |---------------------------*O*--------------------------- > ||_// the State | Ryan Novosielski - [email protected] > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus > || \\ of NJ | Office of Advanced Research Computing - MSB C630, > Newark > `' > -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - [email protected] || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'
