Hello, We are just about to switch to slurm after using torque/maui for quite some time. Our cluster mainly runs MPI jobs. I’m very close to having everything set up and operating as desired but I’m having trouble understanding the prolog behavior. Among other things we want the prolog and epilog scripts to clean up any stray processes. We have the prolog check a few things (e.g. running on an appropreiate file system, etc.) and kill any leftover user processes, echoing the actions to a small banner in stdout. The epilog does some more checks and then cleans up all processes from the users job (using the supplied slurm.epilog.clean script), also echoing some output to stdout. My initial testing was all done with some trivial batch scripts (just some echo statements) in which the prolog only executed on the head node for the job while the epilog executed on all nodes. Because I thought it only executed on the head node, the prolog uses an scontrol command to get all the nodes for the job and then ssh’s to each node to clean up the processes on the other nodes. Here’s an example job script and the output.
[dvicker@service0 slurm]% cat simple.csh #! /bin/csh #SBATCH --tasks-per-node=12 #SBATCH --nodes=2 echo hi [dvicker@service0 slurm]% sbatch simple.csh Submitted batch job 206 [dvicker@service0 slurm]% cat slurm-206.out ------------------ SLURM prolog script on r1i0n0 ------------------ Thu Jan 5 17:09:40 CST 2017 NodeList=r1i0n[0-1] NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* WorkDir=/lustre2/work/dvicker/slurm Cleaning up stray processes on all nodes for this job r1i0n0 r1i0n1 --------------- End of output from prolog script --------------- hi Epilog: Killing all processes from user 'dvicker' on node 'r1i0n0'... Epilog: Killing all processes from user 'dvicker' on node 'r1i0n1'... [dvicker@service0 slurm]% Now I’m testing actual MPI jobs (or jobs that use an srun command) and the prolog is executing on every node. [dvicker@service0 slurm]% cat simple.csh #! /bin/csh #SBATCH --tasks-per-node=12 #SBATCH --nodes=2 srun echo hi [dvicker@service0 slurm]% sbatch simple.csh Submitted batch job 207 [dvicker@service0 slurm]% cat slurm-207.out ------------------ SLURM prolog script on r1i0n0 ------------------ Thu Jan 5 17:14:59 CST 2017 NodeList=r1i0n[0-1] NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* WorkDir=/lustre2/work/dvicker/slurm Cleaning up stray processes on all nodes for this job r1i0n0 r1i0n1 --------------- End of output from prolog script --------------- ------------------ SLURM prolog script on r1i0n1 ------------------ hi Thu Jan 5 17:15:03 CST 2017 hi NodeList=r1i0n[0-1] hi NumNodes=2 NumCPUs=24 NumTasks=24 CPUs/Task=1 ReqB:S:C:T=0:0:*:* hi WorkDir=/lustre2/work/dvicker/slurm hi Cleaning up stray processes on all nodes for this job hi hi hi hi hi hi hi r1i0n0 r1i0n1 --------------- End of output from prolog script --------------- hi hi hi hi hi hi hi hi hi hi hi hi Epilog: Killing all processes from user 'dvicker' on node 'r1i0n0'... Epilog: Killing all processes from user 'dvicker' on node 'r1i0n1'... [dvicker@service0 slurm]% So the srun prolog behavior is not ideal for a couple reasons – the output is too verbose and gets intermingled with the actual job output (the head node seams to execute before the user’s job runs but prolog seems to execute in parallel with the user’s job on the other nodes). The bigger problem is all the nodes ssh-ing to all the other nodes and killing stray processes – this is overkill. It’s also not going to work to have the process clean run in parallel with the user’s job. Is this behavior expected? Is there a way to make the prolog only run on the head node (i.e. the first node of the job)? We’re running slurm version 16.05.7 on RHEL 6. Thanks, Darby
