CUDA_VISBLE_DEVICES is used by many AI framework to determine which gpu to use, like tensorflow. So this environment is critical to us.
-----Original Message----- From: slurm-users <slurm-users-boun...@lists.schedmd.com> On Behalf Of Chris Samuel Sent: Thursday, August 30, 2018 4:42 PM To: slurm-users@lists.schedmd.com Subject: [External] Re: [slurm-users] serious bug about CUDA_VISBLE_DEVICES in the slurm 17.11.7 On Thursday, 30 August 2018 6:38:08 PM AEST Chaofeng Zhang wrote: > The CUDA_VISBLE_DEVICES can't be set NoDevFiles in Slurm 17.11.7. > This is worked when we use Slurm 17.02. You probably should be using cgroups instead to constrain access to GPUs. Then it doesn't matter what you set CUDA_VISBLE_DEVICES to be as processes will only be able to access what they requested. Hope that helps! Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
