Batch job completion failure typically indicates an issue on the slurmd or slurmstepd side of things that slurmctld is unsure how to deal with. Try checking your slurmd logs (debug level) on the impacted nodes. Given the asterisk in the sinfo output, I'm also guessing that slurmd exited. There may be a core file in the same directory as the log, which might be of use understanding the issue.
Doug On May 17, 2017 5:52 AM, "Andy Riebs" <[email protected]> wrote: > > You can use "sinfo -R" to find out why they are being drained -- or at > least a hint, depending on the situation. > > Andy > > On 05/17/2017 08:44 AM, Lennart Karlsson wrote: > >> >> On 05/17/2017 10:08 AM, Baker D.J. wrote: >> >>> Hello, >>> >>> Quite recently I upgraded to Slurm 17.02.2. For some reason every day >>> I've noticed that a number of compute nodes go into draining or drained >>> state after running jobs. For example, this morning I see... >>> >>> [root@blue30 openmpi-1.10.1]# sinfo -Nl | grep drain >>> red0165 1 batch* draining 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> red0691 1 batch* drained 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> red0692 1 batch* draining 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> red0882 1 batch* draining 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> red0895 1 batch* drained 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> red0897 1 batch* drained 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> red0901 1 batch* draining 12 2:6:1 1 0 1 >>> (null) batch job complete f >>> >>> I never saw this behaviour with v16.05.2. Does anyone understand why >>> this is happening, please? Is this a bug introduced in 17.02.2? >>> >>> Best regards, >>> David >>> >> >> Hello David, >> >> We also saw this when we had upgraded to v 17.02, and have the same >> questions. >> >> Cheers, >> -- Lennart Karlsson, UPPMAX, Uppsala University, Sweden >> > > -- > Andy Riebs > [email protected] > Hewlett-Packard Enterprise > High Performance Computing Software Engineering > +1 404 648 9024 > My opinions are not necessarily those of HPE > May the source be with you! >
