Hi, For some reason I got it into my head that the cred_state.new file was on the master. It's not, it's written by slurmd. I'm guessing that what ever was filling up the file system had exited when you looked. When we've encountered this issue before it's usually because someone has kindly dumped loads lots of files in /tmp, and because the location of our SlurmdSpoolDir is on the same file system we encounter the no space on the device message. I would look back through the accounting database and see what was running at the time of the issue. It could be that a batch job filled up the file system.
--- Samuel Gallop From: Tom G [mailto:[email protected]] Sent: 30 January 2017 14:56 To: slurm-dev <[email protected]> Subject: [slurm-dev] RE: nodes randomly can't write to /var/spool/slurmd ? Hi Sam, The master node does not have any disk issues either, plenty of free space and inodes. On Mon, Jan 30, 2017 at 9:40 AM, Sam Gallop (NBI) <[email protected]<mailto:[email protected]>> wrote: Hi, Check the location of the state file directory on the master controller. Is this full? --- Sam Gallop [Description: Macintosh HD:Users:fretter:Documents:SugarSync Shared Folders:NBI:pics for hpc wiki:CiSlogo578x293.png] Have you tried looking through our Documentation Portal<http://docs.cis.nbi.ac.uk/display/CIS/CiSSupport+Home> Our documentation isn’t all text, check out our Video Tutorials<http://docs.cis.nbi.ac.uk/display/CIS/Video+tutorials> Have you tried looking through CiS Service Desk<http://support.cis.nbi.ac.uk/> Keep up to date with availability at CiS Service Status<http://status.cis.nbi.ac.uk/> More information on our HPC, Linux, Storage at HPC Support<http://docs.cis.nbi.ac.uk/display/CIS/CiSSupport+Home> Site To speak to us about technical issues feel free to call the Computing infrastructure for Science team on group phone extension 2003. If your request is urgent, please contact the NBIP Computing Helpdesk at [email protected]<mailto:[email protected]> or call phone extension 1234. From: Tom G [mailto:[email protected]<mailto:[email protected]>] Sent: 30 January 2017 14:38 To: slurm-dev <[email protected]<mailto:[email protected]>> Subject: [slurm-dev] nodes randomly can't write to /var/spool/slurmd ? Hi, A few of the node in my cluster are sporadically going into a DRAIN state, and the last message i see in the slurmd log is: error: write /var/spool/slurmd/cred_state.new error No space left on device However, the node in question has more than enough free space and enough inodes. Has anyone else seen this before? thanks in advance for any help.
