Sites installing the Slurm RPM packages on CentOS 7, RHEL 7, and other
Systemd-based systems will experience a bug in the start-up of the Slurm
daemons. Several sites have made the same discovery, but I would like
to make all affected sites aware of the problem and a suggested workaround.
We discovered that the slurmd daemon didn't set the configured
LimitMEMLOCK ulimits, causing failures in jobs using Infiniband and
OmniPath fabrics. This was tracked down to slurmd being started in
error by the System V style init-script /etc/init.d/slurm. The correct
behavior on Systemd-based systems is to start the slurmd service using
the /usr/lib/systemd/system/slurmd.service as installed by the slurm RPM
package.
The best workaround we have come up with for CentOS/RHEL 7 is to
completely disable the slurm init-script, which was inadvertently
installed, as follows:
chkconfig --del slurm
rm -f /etc/init.d/slurm
This must be repeated every time Slurm is updated.
You must subsequently ensure that the Slurm services are started
correctly using systemctl on the master, database, and compute nodes.
The slurmd.service file correctly sets the ulimits. We have some notes
about this in our Wiki:
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#starting-slurm-daemons-at-boot-time
The above bug is due to the slurm.spec file inadvertently configuring
the init-script in the post-install phase, and it has been reported in
https://bugs.schedmd.com/show_bug.cgi?id=3371. It is expected to be
resolved in Slurm 17.11.
Ole Holm Nielsen
Technical University of Denmark
