Hi, I am trying to catch situations where a job would fail in the prolog script and prevent job execution on that node where that is the case. I have a common section in both prolog and epilog scripts that calls the health check and does `exit 1` if that fails.
After creating a misconfiguration of a node that triggers the check, I
observe the output of the prolog (written to a log file) indicating
that the check tripped. But the job still runs on that node and
slurmctld only notices the failure in the epilog:
[2017-01-20T19:37:04.461] error: job_epilog_complete: JobID=xxxxxx State=0x8005
NodeCnt=1 epilog error on nodexxx, draining the node
So I know my check works, but the failure in the prolog is without
consequence. It is not reported anywhere, apparently. Note that this is
a single-node job only and we have that in slurm.conf:
PrologFlags=Alloc
My envisioned setup also contains
SchedulerParameters=nohold_on_prolog_fail
to make the job run on another node, but I tested both with and without
this flag to make sure. It does not make a difference.
I am using slurm-15.08.13 on the nodes for slurmd and client commands
while the batch server is still on 15.08.12 since I intended to do that
important update without a downtime and already got caught off guard by
the restart of slurmd on the nodes killing/rescheduling all jobs (I
think I remember that jobs did survive restarts of slurmd before …
disturbing on its own and I intend to investigate that, too).
Anyhow, am I observing normal behaviour with the prolog and am I just
misunderstanding some configuration? Is there a bug? I know that there
is the option of calling health checks periodically, too, but I
consider the moment right before starting a job as the right time for
checking if the node is ready for it. I do want to avoid unnecessary
job aborts due to bad timing of the periodic checks and also
unnecessary checks as such while a job is still filling the node and
the check might even interfere.
Alrighty then,
Thomas
--
Dr. Thomas Orgis
Universität Hamburg
RRZ / Basis-Infrastruktur / HPC
Schlüterstr. 70
20146 Hamburg
Tel.: 040/42838 8826
Fax: 040/428 38 6270
smime.p7s
Description: S/MIME cryptographic signature
