Hi, Check the location of the state file directory on the master controller. Is this full?
--- Sam Gallop [Description: Macintosh HD:Users:fretter:Documents:SugarSync Shared Folders:NBI:pics for hpc wiki:CiSlogo578x293.png] Have you tried looking through our Documentation Portal<http://docs.cis.nbi.ac.uk/display/CIS/CiSSupport+Home> Our documentation isn’t all text, check out our Video Tutorials<http://docs.cis.nbi.ac.uk/display/CIS/Video+tutorials> Have you tried looking through CiS Service Desk<http://support.cis.nbi.ac.uk/> Keep up to date with availability at CiS Service Status<http://status.cis.nbi.ac.uk/> More information on our HPC, Linux, Storage at HPC Support<http://docs.cis.nbi.ac.uk/display/CIS/CiSSupport+Home> Site To speak to us about technical issues feel free to call the Computing infrastructure for Science team on group phone extension 2003. If your request is urgent, please contact the NBIP Computing Helpdesk at [email protected]<mailto:[email protected]> or call phone extension 1234. From: Tom G [mailto:[email protected]] Sent: 30 January 2017 14:38 To: slurm-dev <[email protected]> Subject: [slurm-dev] nodes randomly can't write to /var/spool/slurmd ? Hi, A few of the node in my cluster are sporadically going into a DRAIN state, and the last message i see in the slurmd log is: error: write /var/spool/slurmd/cred_state.new error No space left on device However, the node in question has more than enough free space and enough inodes. Has anyone else seen this before? thanks in advance for any help.
