Christopher, Thank you for the tip. That works as expected.
-SS- -----Original Message----- From: slurm-users <slurm-users-boun...@lists.schedmd.com> On Behalf Of Christopher Samuel Sent: Thursday, October 8, 2020 6:52 PM To: slurm-users@lists.schedmd.com Subject: Re: [slurm-users] CUDA environment variable not being set EXTERNAL SENDER On 10/8/20 3:48 pm, Sajesh Singh wrote: > Thank you. Looks like the fix is indeed the missing file > /etc/slurm/cgroup_allowed_devices_file.conf No, you don't want that, that will allow all access to GPUs whether people have requested them or not. What you want is in gres.conf and looks like (hopefully not line wrapped!): NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia0 Cores=0,2,4,6,8 NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia1 Cores=10,12,14,16,18 NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia2 Cores=20,22,24,26,28 NodeName=nodes[01-18] Name=gpu Type=v100 File=/dev/nvidia3 Cores=30,32,34,36,38 All the best, Chris -- Chris Samuel : https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=01%7C01%7Cssingh%40amnh.org%7C19c93fb5353d43eae47f08d86bdcdebd%7Cbe0003e8c6b9496883aeb34586974b76%7C0&sdata=YO9kTd3TSKG6Y2B6NHx%2B59I5rNdZGPESatncINTPC5A%3D&reserved=0 : Berkeley, CA, USA
