Hello, I was hoping some of you may help with this query.
We're running slurm 17.02.1-2. We are using preemption according to https://slurm.schedmd.com/preempt.html. It works great, except that we would like to have different behaviour when a job is canceled by a user, and when a job is canceled and requeued because a higher priority job was submitted. We have some partitions with PreemptMode=CANCEL. However, the documentation says there is no difference. The job basically receives SIGCONT, then SIGTERM, then SIGKILL. The preempt doc says that it sends SIGCONT/SIGTERM twice, but this fact is not enough for us to make a decision. Anybody would have a suggestion? We use sbatch to submit our jobs and I see that requeued jobs have a restart count (SLURM_RESTART_COUNT), but by then we can't use it. Ideally, we would like to receive a different signal when is canceled by a user, and when is canceled because of a higher priority job. I also see that slurmstep writes in stdout/stdout of the job: slurmstepd: error: *** JOB 100 ON ip-172-31-29-32 CANCELLED AT 2017-05-03T15:43:27 *** I just need to be able to access that bit of info within the job, before termination. Any ideas? Thanks in advanced for any tips! -- Giovanni
