Just a comment about cgroups and devices. As I recall (and it has been a long time since I looked at this), cgroups for devices behave differently; to allow a device like GPU 2 out of 4 GPUs on a compute node, you have to "deny" the job GPUs 0, 1 and 3, which seems really backward in order to "allow" the job to use GPU 2.
Since a child process inherits the cgroup of its parent and the slurmd daemon's cgroup has access to all GPUs, the only way it can permit a spawned child process (a job) access to the GPUs not allocated to the job is to deny it those GPUs it cannot use. The Slurm cgroup_allowed_devices_file.conf specifies all the devices every job should have access to, which should not include GPU devices (thus automatically "denying" access to all GPUs), and then Slurm dynamically adds the GPU device(s) the job should have access to when it creates the job's cgroup. By adding the /dev/nvidia* to the file, you have said every job has access to every GPU, which is not what you want. [image: Adaptive Computing] <http://www.adaptivecomputing.com/> [image: Twitter] <http://twitter.com/AdaptiveMoab>[image: LinkedIn] <http://www.linkedin.com/company/448673?goback=.fcs_GLHD_adaptive+computing_false_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2&trk=ncsrch_hits>[image: YouTube] <http://www.youtube.com/adaptivecomputing>[image: GooglePlus] <https://plus.google.com/u/0/102155039310685515037/posts>[image: Facebook] <http://www.facebook.com/pages/Adaptive-Computing/314449798572695?fref=ts>[image: RSS] <http://www.adaptivecomputing.com/feed> Gary D. Brown HPC Product Manager +1 801.717.3700 office +1 801.717.3738 fax +1 801.717.3382 desk 1712 S. East Bay Blvd, Suite 300 Provo, UT 84606 USA www.adaptivecomputing.com On Thu, Aug 11, 2016 at 11:05 AM, Kilian Cavalotti < [email protected]> wrote: > > Hi Ryan, > > You probably shouldn't have /dev/nvidia* devices listed in > cgroup_allowed_devices_file.conf, Slurm will automatically manage them > and add them to the list of authorized devices in a cgroup when a job > starts. For reference, our cgroup_allowed_devices_file.conf contains > this: > > /dev/null > /dev/urandom > /dev/zero > /dev/sda* > /dev/cpu/*/* > /dev/pts/* > /dev/tty* > > Also, make sure you have the correct Gres definitions for your nodes > in slurm.conf: > > NodeName=foobar Gres=gpu:4 ... > > and that you have a matching gres.conf, with: > > NodeName=foobar Name=gpu File=/dev/nvidia[0-3] > > > Now if that still doesn't work, I'd recommend adding DebugFlags=Gres > to your slurm.conf and look at what the Slurm controller logs. > > Cheers, > -- > Kilian >
