Hi there,
I am using Slurm version 14.11.9 on a cluster.
These days I am launching jobs based on sbatch allocation for one single 24
core-node and then running serial code in background as job steps:
$ sbatch -N1 --exclusive -J OPTIM -o log myscript.sh
$ cat myscript
#!/bin/bash
for i in $(1..24);do
srun -N1 -n1 --exclusive --mem=10 -J code code.exe &
done
wait
To avoid long listing with info of every step, I used to monitor the
succesive jobs collapsing sacct with -X option:
$ sacct --name=OPTIM -X
JobID JobName AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ----------
5434321 OPTIM 24 COMPLETED 0:0
5434327 OPTIM 24 COMPLETED 0:0
5434333 OPTIM 24 COMPLETED 0:0
......
I was assuming that those ExitCode were a reliable indication of a
successful run.
However, after noticed that one of the runs took a little longer than usual
I observed that one of its steps appeared as CANCELLED:
$ sacct -j 5434327
JobID JobName AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ----------
5434327 OPTIM 24 COMPLETED 0:0
5434327.bat+ batch 24 COMPLETED 0:0
5434327.0 code 1 COMPLETED 0:0
5434327.1 code 1 COMPLETED 0:0
5434327.2 code 1 COMPLETED 0:0
*5434327.3 **code ** 1 CANCELLED -2:0*
5434327.4 code 1 COMPLETED 0:0
5434327.5 code 1 COMPLETED 0:0
5434327.6 code 1 COMPLETED 0:0
....
I thought that a failure in any step would be reflected in State and
ExitCode of the first two entries of sacct (allocation and batch step).
Is it not reliable using sacct -X to check multi-step jobs?
How can be interpreted the successful exit code of the first two entries
after one step failed?
Regards,
Nigella
