Andy is right. When you restart the slurmd daemon, it inherits the system limits from your login session, which are different from the default system limits when the daemon is started on boot-up.
If you modified /etc/security/limits.conf, or made changes in any of the bash startup scripts to improve the user environment, you should ensure that those same changes are added to /etc/sysconfig/slurm so that they can be applied to the slurmd daemons on boot-up. Regards, --Chris -----Original Message----- From: Riebs, Andy Sent: Monday, August 25, 2014 6:24 AM To: slurm-dev Subject: [slurm-dev] Re: Intel MPI Performance inconsistency (and workaround) Assuming this is a Gnu/Linux system, be sure that you have /etc/sysconfig/slurm on all nodes with the line ulimit -l unlimited That can account for differences in processing between system startup and subsequently restarting the daemons by hand. Andy On 08/21/2014 02:42 PM, Jesse Stroik wrote: > > Slurmites, > > We recently noticed sporadic performance inconsistencies on one of our > clusters. We discovered that if we restarted slurmd in an interactive > shell, we observed correct performance. > > To track down the cause, we ran: > > (1) single-node linpack > (2) dual node mp_linpack > (3) mpptest > > On affected nodes, Linpack performance was normal and mp_linpack was > about 85% as high as expected. > > mpptest, which measures MPI performance, was our smoking gun. > Latencies to be 10x higher than expected (~20us instead of < 2us). We > were able to consistently reproduce the issue with freshly imaged or > freshly rebooted nodes. Upon restarting slurmd on each execution node > manually, MPI latencies immediately improved to the expected < 2us for > our set of tested nodes. > > The cluster is under fairly heavy use right now so we don't have the > luxury of diagnosing this thoroughly and determining the cause. We > wanted to share this experience with others in case it can help other > users or if any slurm developers would like us to file a bug report > and be interested in gathering further information. > > Best, > Jesse Stroik > > University of Wisconsin
