Hi Darby, I think the discrepency of behaviour you're seeing in the prolog is because of this:
NOTE: By default the Prolog script is ONLY run on any individual node when it first sees a job step from a new allocation; it does not run the Prolog immediately when an allocation is granted. If no job steps from an allocation are run on a node, it will never run the Prolog for that allocation. This Prolog behaviour can be changed by the PrologFlags parameter. The Epilog, on the other hand, always runs on every node of an allocation when the allocation is released. If you really want to run something only on the 'first node of the job' then you might be able to cobble something together using PrologSlurmctld instead of Prolog, but that would involve parsing the SLURM_JOB_NODELIST variable. But I'd recommend not using the prolog for attempting to kill stray user processes; as you note yourself it's more than likely error-prone, the order of execution is non-deterministic, and it's all a bit messy. Why not just use the Epilog (e.g. the supplied slurm.epilog.clean script) to clean up a node after the job has finished. As noted in the man page above, the Epilog will always run on every node of the allocation. There will be no more stray user processes when the job has finished, so any clean-up in a prolog would have nothing to do. Paddy On Thu, Jan 05, 2017 at 08:07:57PM -0800, Vicker, Darby (JSC-EG311) wrote: > Hello, > > We are just about to switch to slurm after using torque/maui for quite some > time. Our cluster mainly runs MPI jobs. I???m very close to having > everything set up and operating as desired but I???m having trouble > understanding the prolog behavior. Among other things we want the prolog and > epilog scripts to clean up any stray processes. We have the prolog check a > few things (e.g. running on an appropreiate file system, etc.) and kill any > leftover user processes, echoing the actions to a small banner in stdout. > The epilog does some more checks and then cleans up all processes from the > users job (using the supplied slurm.epilog.clean script), also echoing some > output to stdout. My initial testing was all done with some trivial batch > scripts (just some echo statements) in which the prolog only executed on the > head node for the job while the epilog executed on all nodes. Because I > thought it only executed on the head node, the prolog uses an scontrol > command to get all the nodes for the job and then ssh???s to each node to > clean up the processes on the other nodes. Here???s an example job script > and the output. > > [dvicker@service0 slurm]% cat simple.csh > #! /bin/csh > #SBATCH --tasks-per-node=12 > #SBATCH --nodes=2 > > echo hi > [dvicker@service0 slurm]% sbatch simple.csh > Submitted batch job 206 > [dvicker@service0 slurm]% cat slurm-206.out > ------------------ SLURM prolog script on r1i0n0 ------------------ > Thu Jan 5 17:09:40 CST 2017 > NodeList=r1i0n[0-1] > NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > WorkDir=/lustre2/work/dvicker/slurm > Cleaning up stray processes on all nodes for this job > r1i0n0 > r1i0n1 > --------------- End of output from prolog script --------------- > hi > Epilog: Killing all processes from user 'dvicker' on node 'r1i0n0'... > Epilog: Killing all processes from user 'dvicker' on node 'r1i0n1'... > [dvicker@service0 slurm]% > > > > > Now I???m testing actual MPI jobs (or jobs that use an srun command) and the > prolog is executing on every node. > > > > > [dvicker@service0 slurm]% cat simple.csh > #! /bin/csh > #SBATCH --tasks-per-node=12 > #SBATCH --nodes=2 > > srun echo hi > [dvicker@service0 slurm]% sbatch simple.csh > Submitted batch job 207 > [dvicker@service0 slurm]% cat slurm-207.out > ------------------ SLURM prolog script on r1i0n0 ------------------ > Thu Jan 5 17:14:59 CST 2017 > NodeList=r1i0n[0-1] > NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > WorkDir=/lustre2/work/dvicker/slurm > Cleaning up stray processes on all nodes for this job > r1i0n0 > r1i0n1 > --------------- End of output from prolog script --------------- > ------------------ SLURM prolog script on r1i0n1 ------------------ > hi > Thu Jan 5 17:15:03 CST 2017 > hi > NodeList=r1i0n[0-1] > hi > NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* > hi > WorkDir=/lustre2/work/dvicker/slurm > hi > Cleaning up stray processes on all nodes for this job > hi > hi > hi > hi > hi > hi > hi > r1i0n0 > r1i0n1 > --------------- End of output from prolog script --------------- > hi > hi > hi > hi > hi > hi > hi > hi > hi > hi > hi > hi > Epilog: Killing all processes from user 'dvicker' on node 'r1i0n0'... > Epilog: Killing all processes from user 'dvicker' on node 'r1i0n1'... > [dvicker@service0 slurm]% > > > > > > So the srun prolog behavior is not ideal for a couple reasons ??? the output > is too verbose and gets intermingled with the actual job output (the head > node seams to execute before the user???s job runs but prolog seems to > execute in parallel with the user???s job on the other nodes). The bigger > problem is all the nodes ssh-ing to all the other nodes and killing stray > processes ??? this is overkill. It???s also not going to work to have the > process clean run in parallel with the user???s job. > > Is this behavior expected? Is there a way to make the prolog only run on the > head node (i.e. the first node of the job)? We???re running slurm version > 16.05.7 on RHEL 6. > > Thanks, > Darby > > -- Paddy Doyle Trinity Centre for High Performance Computing, Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. Phone: +353-1-896-3725 http://www.tchpc.tcd.ie/
