Hello! We have a huge number of jobs stuck in CG state from a user who probably wrote code with bad I/O. "scancel" does not make them go away. Is there a way for admins to get rid of these jobs without draining and rebooting the nodes. I read somewhere that killing the respective slurmstepd process will do the job. Is this possible? Any other solutions? Also are there any parameters in slurm.conf one can set to manage such situations better?
Best, Durai MPI Tübingen
