The CUDA_VISBLE_DEVICES can't be set NoDevFiles in Slurm 17.11.7. This is worked when we use Slurm 17.02.
Slurm 17.02: [root@head ~]# export CUDA_VISIBLE_DEVICES=0,1 [root@head ~]# srun -N1 -n1 --gres=none --nodelist=head /usr/bin/env|grep CUDA CUDA_HOME=/usr/local/cuda CUDA_VISIBLE_DEVICES=NoDevFiles Slurm 17.11.7: [root@head ~]# export CUDA_VISIBLE_DEVICES=0,1 [root@head ~]# srun -N1 -n1 --gres=none --nodelist=head /usr/bin/env|grep CUDA CUDA_HOME=/usr/local/cuda CUDA_VISIBLE_DEVICES=0,1 Even it is not work pass env through export. [root@head ~]# export CUDA_VISIBLE_DEVICES=0,1 [root@head ~]# srun -N1 -n1 --gres=none --nodelist=head --export=All,CUDA_VISIBLE_DEVICES=NoDevFiles /usr/bin/env|grep CUDA CUDA_HOME=/usr/local/cuda CUDA_VISIBLE_DEVICES=0,1
