Hi Ryan, You probably shouldn't have /dev/nvidia* devices listed in cgroup_allowed_devices_file.conf, Slurm will automatically manage them and add them to the list of authorized devices in a cgroup when a job starts. For reference, our cgroup_allowed_devices_file.conf contains this:
/dev/null /dev/urandom /dev/zero /dev/sda* /dev/cpu/*/* /dev/pts/* /dev/tty* Also, make sure you have the correct Gres definitions for your nodes in slurm.conf: NodeName=foobar Gres=gpu:4 ... and that you have a matching gres.conf, with: NodeName=foobar Name=gpu File=/dev/nvidia[0-3] Now if that still doesn't work, I'd recommend adding DebugFlags=Gres to your slurm.conf and look at what the Slurm controller logs. Cheers, -- Kilian
