CUDA_VISIBLE_DEVICES=0 on both jobs. This is also true for the any other jobs that work just fine.
There are only one device visible in the job. /Magnus On 2019-01-15 13:28, arnaud.ren...@univ-reims.fr wrote:
What is the value of env. variable CUDA_VISIBLE_DEVICES ? Envoyé depuis mon mobile Huawei -------- Message original -------- Objet : [slurm-users] Two jobs ends up on one GPU? De : Magnus Jonsson À : slurm-users@lists.schedmd.com Cc : Hi! We have machines with multiple GPUs (Nvidia V100). We allow multiple (two) jobs on the nodes. We have a user that in some way have managed to get both jobs to end up on the same GPU (verified via nvidia-smi). We are using cgroups and the nvidia-smi command only shows one of the GPUs (if only one GPU are requested) and only the defined /dev/nvidia? device are accessable. We are unable to reproduce this. Have anybody seen anything like this? /Magnus-- Magnus Jonsson, Developer, HPC2N, Umeå Universitet
-- Magnus Jonsson, Developer, HPC2N, Umeå Universitet
