Yes, that it. Thanks. I see that node in “man slurm.conf” now. FYI for the
slurm devs, it would be nice to add this to
https://slurm.schedmd.com/prolog_epilog.html too - I tend to use that as my
main reference.
I tried using PrologSlurmctld but since that runs as the slurm user (not root)
you can’t do process clean up or echo things into the user’s stdout file. It
wasn’t too bad to detect the first node in the job to limit the prolog actions.
nodes=`/usr/bin/scontrol show hostnames | xargs` # List of nodes for this
job
headnode=`echo $nodes | awk '{print $1}'` # First node for this job
if [ $headnode == $SLURMD_NODENAME ] ; then
.
.
.
fi
In the past, I’ve had the epilog clean up script fail to clean all the stray
processes. By doing another round of clean up in the prolog it just gives the
current job the best chance of getting an idle set of nodes. So I’m looping
over $nodes in the if test above, which seems to be working fine. We allocate
all nodes to a job (SelectType=select/linear) so it is fairly safe to do this.
Thanks for the help.
Darby
-----Original Message-----
From: Paddy Doyle <[email protected]>
Reply-To: slurm-dev <[email protected]>
Date: Friday, January 6, 2017 at 6:39 AM
To: slurm-dev <[email protected]>
Subject: [slurm-dev] Re: Prolog behavior with and without srun
Hi Darby,
I think the discrepency of behaviour you're seeing in the prolog is because
of
this:
NOTE: By default the Prolog script is ONLY run on any individual node
when it first sees a job step from a new allocation; it does not run
the Prolog immediately when an allocation is granted. If no job steps
from an allocation are run on a node, it will never run the Prolog for
that allocation. This Prolog behaviour can be changed by the PrologFlags
parameter. The Epilog, on the other hand, always runs on every node of
an allocation when the allocation is released.
If you really want to run something only on the 'first node of the job'
then you
might be able to cobble something together using PrologSlurmctld instead of
Prolog, but that would involve parsing the SLURM_JOB_NODELIST variable.
But I'd recommend not using the prolog for attempting to kill stray user
processes; as you note yourself it's more than likely error-prone, the order
of execution is non-deterministic, and it's all a bit messy.
Why not just use the Epilog (e.g. the supplied slurm.epilog.clean script) to
clean up a node after the job has finished. As noted in the man page above,
the
Epilog will always run on every node of the allocation. There will be no
more
stray user processes when the job has finished, so any clean-up in a prolog
would have nothing to do.
Paddy
On Thu, Jan 05, 2017 at 08:07:57PM -0800, Vicker, Darby (JSC-EG311) wrote:
> Hello,
>
> We are just about to switch to slurm after using torque/maui for quite
some time. Our cluster mainly runs MPI jobs. I???m very close to having
everything set up and operating as desired but I???m having trouble
understanding the prolog behavior. Among other things we want the prolog and
epilog scripts to clean up any stray processes. We have the prolog check a few
things (e.g. running on an appropreiate file system, etc.) and kill any
leftover user processes, echoing the actions to a small banner in stdout. The
epilog does some more checks and then cleans up all processes from the users
job (using the supplied slurm.epilog.clean script), also echoing some output to
stdout. My initial testing was all done with some trivial batch scripts (just
some echo statements) in which the prolog only executed on the head node for
the job while the epilog executed on all nodes. Because I thought it only
executed on the head node, the prolog uses an scontrol command to get all the
nodes for the job and then ssh???s to each node to clean up the processes on
the other nodes. Here???s an example job script and the output.
>
> [dvicker@service0 slurm]% cat simple.csh
> #! /bin/csh
> #SBATCH --tasks-per-node=12
> #SBATCH --nodes=2
>
> echo hi
> [dvicker@service0 slurm]% sbatch simple.csh
> Submitted batch job 206
> [dvicker@service0 slurm]% cat slurm-206.out
> ------------------ SLURM prolog script on r1i0n0 ------------------
> Thu Jan 5 17:09:40 CST 2017
> NodeList=r1i0n[0-1]
> NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> WorkDir=/lustre2/work/dvicker/slurm
> Cleaning up stray processes on all nodes for this job
> r1i0n0
> r1i0n1
> --------------- End of output from prolog script ---------------
> hi
> Epilog: Killing all processes from user 'dvicker' on node 'r1i0n0'...
> Epilog: Killing all processes from user 'dvicker' on node 'r1i0n1'...
> [dvicker@service0 slurm]%
>
>
>
>
> Now I???m testing actual MPI jobs (or jobs that use an srun command) and
the prolog is executing on every node.
>
>
>
>
> [dvicker@service0 slurm]% cat simple.csh
> #! /bin/csh
> #SBATCH --tasks-per-node=12
> #SBATCH --nodes=2
>
> srun echo hi
> [dvicker@service0 slurm]% sbatch simple.csh
> Submitted batch job 207
> [dvicker@service0 slurm]% cat slurm-207.out
> ------------------ SLURM prolog script on r1i0n0 ------------------
> Thu Jan 5 17:14:59 CST 2017
> NodeList=r1i0n[0-1]
> NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> WorkDir=/lustre2/work/dvicker/slurm
> Cleaning up stray processes on all nodes for this job
> r1i0n0
> r1i0n1
> --------------- End of output from prolog script ---------------
> ------------------ SLURM prolog script on r1i0n1 ------------------
> hi
> Thu Jan 5 17:15:03 CST 2017
> hi
> NodeList=r1i0n[0-1]
> hi
> NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
> hi
> WorkDir=/lustre2/work/dvicker/slurm
> hi
> Cleaning up stray processes on all nodes for this job
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> r1i0n0
> r1i0n1
> --------------- End of output from prolog script ---------------
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> hi
> Epilog: Killing all processes from user 'dvicker' on node 'r1i0n0'...
> Epilog: Killing all processes from user 'dvicker' on node 'r1i0n1'...
> [dvicker@service0 slurm]%
>
>
>
>
>
> So the srun prolog behavior is not ideal for a couple reasons ??? the
output is too verbose and gets intermingled with the actual job output (the
head node seams to execute before the user???s job runs but prolog seems to
execute in parallel with the user???s job on the other nodes). The bigger
problem is all the nodes ssh-ing to all the other nodes and killing stray
processes ??? this is overkill. It???s also not going to work to have the
process clean run in parallel with the user???s job.
>
> Is this behavior expected? Is there a way to make the prolog only run on
the head node (i.e. the first node of the job)? We???re running slurm version
16.05.7 on RHEL 6.
>
> Thanks,
> Darby
>
>
--
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/
