Just to eliminate some part of the pty process as the problem, I pulled the pty and changed the command from “bash -i” to “hostname”. Same hang.
> On Aug 11, 2016, at 2:31 PM, Ryan Novosielski <[email protected]> wrote: > > I’m also noticing that I sometimes see this after a few minutes: > > [novosirj@fen2 ~]$ srun -n1 -p tesla --gres=gpu:1 --reservation=cgroups_test > --pty --time=20:00 bash -i > srun: error: pty: accept failure: Interrupted system call > > But things work fine without the --gres definition. > > I’m including an excerpt of the DebugFlags=Gres output here. Much of this is > repeated every minute or so while the job is running: > > [2016-08-11T14:27:42.586] gres: gpu state for job 5377696 > [2016-08-11T14:27:42.586] gres_cnt:1 node_cnt:0 type:(null) > [2016-08-11T14:27:42.586] gres/gpu: state for gpu002 > [2016-08-11T14:27:42.586] gres_cnt found:8 configured:8 avail:8 alloc:1 > [2016-08-11T14:27:42.586] gres_bit_alloc:0 > [2016-08-11T14:27:42.586] gres_used:(null) > [2016-08-11T14:27:42.586] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:42.586] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:42.586] topo_gres_cnt_alloc[0]:1 > [2016-08-11T14:27:42.586] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:42.586] type[0]:(null) > [2016-08-11T14:27:42.586] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:42.587] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:42.587] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:42.587] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:42.587] type[1]:(null) > [2016-08-11T14:27:42.587] gres/mic: state for gpu002 > [2016-08-11T14:27:42.587] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:42.587] gres_bit_alloc:NULL > [2016-08-11T14:27:42.587] gres_used:(null) > [2016-08-11T14:27:42.587] gres: gpu state for job 5377696 > [2016-08-11T14:27:42.587] gres_cnt:1 node_cnt:1 type:(null) > [2016-08-11T14:27:42.588] gres_bit_step_alloc:NULL > [2016-08-11T14:27:42.588] gres_bit_alloc[0]:0 > [2016-08-11T14:27:42.588] gres_cnt_step_alloc[0]:0 > [2016-08-11T14:27:42.588] sched: _slurm_rpc_allocate_resources JobId=5377696 > NodeList=gpu002 usec=2968 > [2016-08-11T14:27:42.593] gres/gpu state for step 5377696.0 > [2016-08-11T14:27:42.593] gres_cnt:1 node_cnt:0 type:(null) > [2016-08-11T14:27:42.593] node_in_use:NULL > [2016-08-11T14:27:42.593] gres/gpu state for step 5377696.0 > [2016-08-11T14:27:42.593] gres_cnt:1 node_cnt:1 type:(null) > [2016-08-11T14:27:42.593] gres_bit_alloc[0]:0 > [2016-08-11T14:27:48.161] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.161] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.161] gres_bit_alloc: > [2016-08-11T14:27:48.161] gres_used:(null) > [2016-08-11T14:27:48.161] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.161] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.161] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.161] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.161] type[0]:(null) > [2016-08-11T14:27:48.162] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.162] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.162] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.162] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.162] type[1]:(null) > [2016-08-11T14:27:48.162] gres/mic: state for gpu002 > [2016-08-11T14:27:48.162] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.162] gres_bit_alloc:NULL > [2016-08-11T14:27:48.162] gres_used:(null) > [2016-08-11T14:27:48.162] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.162] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.162] gres_bit_alloc: > [2016-08-11T14:27:48.162] gres_used:(null) > [2016-08-11T14:27:48.162] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.162] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.162] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.162] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.162] type[0]:(null) > [2016-08-11T14:27:48.162] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.162] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.162] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.162] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.163] type[1]:(null) > [2016-08-11T14:27:48.163] gres/mic: state for gpu002 > [2016-08-11T14:27:48.163] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.163] gres_bit_alloc:NULL > [2016-08-11T14:27:48.163] gres_used:(null) > [2016-08-11T14:27:48.163] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.163] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.163] gres_bit_alloc: > [2016-08-11T14:27:48.163] gres_used:(null) > [2016-08-11T14:27:48.163] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.163] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.163] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.163] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.163] type[0]:(null) > [2016-08-11T14:27:48.163] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.163] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.163] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.163] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.163] type[1]:(null) > [2016-08-11T14:27:48.163] gres/mic: state for gpu002 > [2016-08-11T14:27:48.163] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.163] gres_bit_alloc:NULL > [2016-08-11T14:27:48.163] gres_used:(null) > [2016-08-11T14:27:48.164] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.164] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.164] gres_bit_alloc: > [2016-08-11T14:27:48.164] gres_used:(null) > [2016-08-11T14:27:48.164] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.164] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.164] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.164] type[0]:(null) > [2016-08-11T14:27:48.164] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.164] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.164] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.164] type[1]:(null) > [2016-08-11T14:27:48.164] gres/mic: state for gpu002 > [2016-08-11T14:27:48.164] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.164] gres_bit_alloc:NULL > [2016-08-11T14:27:48.164] gres_used:(null) > [2016-08-11T14:27:48.164] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.164] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.164] gres_bit_alloc: > [2016-08-11T14:27:48.164] gres_used:(null) > [2016-08-11T14:27:48.164] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.164] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.164] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.164] type[0]:(null) > [2016-08-11T14:27:48.164] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.164] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.164] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.165] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.165] type[1]:(null) > [2016-08-11T14:27:48.165] gres/mic: state for gpu002 > [2016-08-11T14:27:48.165] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.165] gres_bit_alloc:NULL > [2016-08-11T14:27:48.165] gres_used:(null) > [2016-08-11T14:27:48.165] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.165] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.165] gres_bit_alloc: > [2016-08-11T14:27:48.165] gres_used:(null) > [2016-08-11T14:27:48.165] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.165] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.165] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.165] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.165] type[0]:(null) > [2016-08-11T14:27:48.165] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.165] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.165] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.165] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.165] type[1]:(null) > [2016-08-11T14:27:48.165] gres/mic: state for gpu002 > [2016-08-11T14:27:48.165] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.165] gres_bit_alloc:NULL > [2016-08-11T14:27:48.165] gres_used:(null) > [2016-08-11T14:27:48.166] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.166] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.166] gres_bit_alloc: > [2016-08-11T14:27:48.166] gres_used:(null) > [2016-08-11T14:27:48.166] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.166] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.166] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.166] type[0]:(null) > [2016-08-11T14:27:48.166] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.166] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.166] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.166] type[1]:(null) > [2016-08-11T14:27:48.166] gres/mic: state for gpu002 > [2016-08-11T14:27:48.166] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.166] gres_bit_alloc:NULL > [2016-08-11T14:27:48.166] gres_used:(null) > [2016-08-11T14:27:48.166] gres/gpu: state for gpu002 > [2016-08-11T14:27:48.166] gres_cnt found:8 configured:8 avail:8 alloc:0 > [2016-08-11T14:27:48.166] gres_bit_alloc: > [2016-08-11T14:27:48.166] gres_used:(null) > [2016-08-11T14:27:48.166] topo_cpus_bitmap[0]:0-5 > [2016-08-11T14:27:48.166] topo_gres_bitmap[0]:0-3 > [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[0]:0 > [2016-08-11T14:27:48.166] topo_gres_cnt_avail[0]:4 > [2016-08-11T14:27:48.166] type[0]:(null) > [2016-08-11T14:27:48.166] topo_cpus_bitmap[1]:6-11 > [2016-08-11T14:27:48.166] topo_gres_bitmap[1]:4-7 > [2016-08-11T14:27:48.166] topo_gres_cnt_alloc[1]:0 > [2016-08-11T14:27:48.166] topo_gres_cnt_avail[1]:4 > [2016-08-11T14:27:48.166] type[1]:(null) > [2016-08-11T14:27:48.166] gres/mic: state for gpu002 > [2016-08-11T14:27:48.166] gres_cnt found:0 configured:0 avail:0 alloc:0 > [2016-08-11T14:27:48.166] gres_bit_alloc:NULL > [2016-08-11T14:27:48.166] gres_used:(null) > >> On Aug 11, 2016, at 2:11 PM, Ryan Novosielski <[email protected]> wrote: >> >> Thanks very much for your reply — that part of the documentation was very >> unclear, unfortunately. No GRES devices were ever mentioned in the setup, >> but it seemed to imply that you’d need to know all of the devices that are >> going to be used. I’ve now removed the GRES devices (I’m curious — what >> about Phi/MIC? Same thing, I guess?). >> >> I have an otherwise-working system using GRES devices (checking out GPUs, >> etc.) — the only change was to enable cgroups. All of my node definitions >> are OK: >> >> [root@fen2 novosirj]# scontrol show node gpu002 >> NodeName=gpu002 Arch=x86_64 CoresPerSocket=6 >> CPUAlloc=1 CPUErr=0 CPUTot=12 CPULoad=0.43 Features=sandybrige >> Gres=gpu:8 >> >> …and… >> >> [root@gpu002 ~]# more /etc/slurm/gres.conf >> # 8 x Tesla M2070 >> NodeName=gpu[002] Name=gpu File=/dev/nvidia[0-3] CPUs=0-5 >> NodeName=gpu[002] Name=gpu File=/dev/nvidia[4-7] CPUs=6-11 >> >> I’m still seeing the same hang after removing /dev/nvidia*. A couple of >> things though — there /was/ something in slurmd.log that I think may be >> relevant (somehow thought it was unassociated with jobs before): >> >> [2016-08-11T14:09:42.524] launch task 5377692.0 request from >> [email protected] (port 10985) >> [2016-08-11T14:09:42.525] error: gres/mic unable to set OFFLOAD_DEVICES, no >> device files configured >> [2016-08-11T14:09:42.525] _run_prolog: run job script took usec=4 >> [2016-08-11T14:09:42.525] _run_prolog: prolog with lock for job 5377692 ran >> for 0 seconds >> [2016-08-11T14:09:42.530] [5377692.0] gres/mic unable to set >> OFFLOAD_DEVICES, no device files configured >> [2016-08-11T14:09:42.531] [5377692.0] gpu 0 is device number 0 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 1 is device number 1 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 2 is device number 2 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 3 is device number 3 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 4 is device number 4 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 5 is device number 5 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 6 is device number 6 >> [2016-08-11T14:09:42.531] [5377692.0] gpu 7 is device number 7 >> [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: >> /slurm/uid_109366/job_5377692: alloc=64391MB mem.limit=63875MB >> memsw.limit=unlimited >> [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: >> /slurm/uid_109366/job_5377692/step_0: alloc=64391MB mem.limit=63875MB >> memsw.limit=unlimited >> [2016-08-11T14:09:42.533] [5377692.0] task/cgroup: manage devices jor job >> '5377692' >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:3 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:9 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 1:5 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:8 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:7 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:6 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:5 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:2 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:1 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device b 8:0 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:11 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:11 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:10 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:10 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:9 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:9 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:8 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:8 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:7 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:7 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:6 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:6 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:5 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:5 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:4 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:4 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:3 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:3 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:2 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:2 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:1 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:1 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 203:0 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 202:0 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c >> 136:0 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Default access allowed to device c 5:2 >> rwm >> [2016-08-11T14:09:42.534] [5377692.0] Allowing access to device 819275 rwm >> [2016-08-11T14:09:42.534] [5377692.0] Not allowing access to device 819275 >> rwm819275 rwm >> >> I’ll try adding the Gres debugging, but is there some way to figure out what >> this alleged device “819275” is (this number will change with each job). >> >>> On Aug 11, 2016, at 1:06 PM, Kilian Cavalotti >>> <[email protected]> wrote: >>> >>> >>> Hi Ryan, >>> >>> You probably shouldn't have /dev/nvidia* devices listed in >>> cgroup_allowed_devices_file.conf, Slurm will automatically manage them >>> and add them to the list of authorized devices in a cgroup when a job >>> starts. For reference, our cgroup_allowed_devices_file.conf contains >>> this: >>> >>> /dev/null >>> /dev/urandom >>> /dev/zero >>> /dev/sda* >>> /dev/cpu/*/* >>> /dev/pts/* >>> /dev/tty* >>> >>> Also, make sure you have the correct Gres definitions for your nodes >>> in slurm.conf: >>> >>> NodeName=foobar Gres=gpu:4 ... >>> >>> and that you have a matching gres.conf, with: >>> >>> NodeName=foobar Name=gpu File=/dev/nvidia[0-3] >>> >>> >>> Now if that still doesn't work, I'd recommend adding DebugFlags=Gres >>> to your slurm.conf and look at what the Slurm controller logs. >>> >>> Cheers, >>> -- >>> Kilian >>> >> >> -- >> ____ >> || \\UTGERS, >> |---------------------------*O*--------------------------- >> ||_// the State | Ryan Novosielski - [email protected] >> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus >> || \\ of NJ | Office of Advanced Research Computing - MSB C630, >> Newark >> `' >> > > -- > ____ > || \\UTGERS, > |---------------------------*O*--------------------------- > ||_// the State | Ryan Novosielski - [email protected] > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus > || \\ of NJ | Office of Advanced Research Computing - MSB C630, > Newark > `' > -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - [email protected] || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'
