We are setting up preemption using QOS on our cluster. Documentation seems to say that when a job is preempted it should be getting a SIGCONT and SOGTERM when selected for preemption, and then a SIGTERM, SIGCONT, AND SIGKILL at the end of gracetime.
We have checked all of this and we are sent the signals at the end of GraceTime, but not when selected for preemption. We are listening for these signals to checkpoint when preempted. We are checking for the signals both in the script and the launched executables in case we are wrong about what catches the signals. I am unclear whether the problem is of my understanding of how it is supposed to work, my configuration, or the documentation. The doc that mentions the signals sent it http://slurm.schedmd.com/preempt.html. This is the qos setup: Name GraceTime Preempt PreemptMode ---------- ---------- ---------- ----------- normal 00:00:00 cluster sxs-lo 00:20:00 cancel sxs-hi 00:20:00 sxs-lo cancel /etc/slurm/slurm.conf: … PreemptType=preempt/qos PreemptMode=CANCEL … What am I doing wrong on this? Thanks, Naveed
