I'm testing an installation of SLURM that is newly-configured in the following way:
ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup JobAcctGatherType=jobacct_gather/cgroup I'm having good luck submitting jobs with both memory and CPU limits -- jobs are cancelled for violating cgroup memory limits, and jobs that run are isolated to the cores they are assigned. When I got to testing for GPU cards, I'm running into trouble. Adding only --gres=gpu:1 to the srun command line that otherwise gives me a bash on the node (srun -n1 -p tesla --pty bash -i) now hangs (you can't even CTRL-C out of it). I'm not seeing anything interesting about it in either the slurmctld log file or the slurmd logfile on the node. Any idea where I should start to look? Also, it's not really clear what I should have in cgroup_allowed_devices_file.conf for GPUs. I currently have /dev/nvidia* -- this covers everything including "nvidia" in the name. Are there maybe others that I'm missing? Let me know what would be helpful info to provide. Thank you! -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - [email protected] || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `'
