Hi Loris > We do something similar. At the end of our script pointed to by > EpilogSlurmctld we have
Using EpilogSlurmctld only works if the slurmctld user is root (or slurm with root privileges), right? I opted for the normal Epilog since we wanted to avoid running slurm as root and I don’t have to worry about ownership of the output file. Sebastian > Am 16.09.2022 um 09:09 schrieb Loris Bennett <loris.benn...@fu-berlin.de>: > > Hi Hermann, > > Sebastian Potthoff <s.potth...@uni-muenster.de > <mailto:s.potth...@uni-muenster.de>> writes: > >> Hi Hermann, >> >> I happened to read along this conversation and was just solving this issue >> today. I added this part to the epilog script to make it work: >> >> # Add job report to stdout >> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | >> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }') >> >> NODELIST=($(/usr/bin/scontrol show hostnames)) >> >> # Only add to StdOut file if it exists and if we are the first node >> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ] >> then >> echo "################################# JOB REPORT >> ##################################" >> $StdOut >> /usr/bin/seff $SLURM_JOB_ID >> $StdOut >> echo >> "###############################################################################" >> >> $StdOut >> fi > > We do something similar. At the end of our script pointed to by > EpilogSlurmctld we have > > OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'` > if [ ! -f "$OUT" ]; then > exit > fi > > printf "\n== Epilog Slurmctld > ==================================================\n\n" >> ${OUT} > > seff ${SLURM_JOB_ID} >> ${OUT} > > printf > "\n======================================================================\n" > >> ${OUT} > > chown ${user} ${OUT} > > Cheers, > > Loris > >> Contrary to what it says in the slurm docs >> https://slurm.schedmd.com/prolog_epilog.html I was not able to use the env >> var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I had to >> make sure it is only called by the „leading“ node as the epilog script will >> be called by ALL nodes of a multinode job and they would all call seff and >> clutter up the output. Last thing was to check if StdOut is >> not of length zero (i.e. it exists). Interactive jobs would otherwise cause >> the node to drain. >> >> Maybe this helps. >> >> Kind regards >> Sebastian >> >> PS: goslmailer looks quite nice with its recommendations! Will definitely >> look into it. >> >> -- >> Westfälische Wilhelms-Universität (WWU) Münster >> WWU IT >> Sebastian Potthoff (eScience / HPC) >> >> Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler >> <hermann.schwaerz...@uibk.ac.at>: >> >> Hi Ole, >> >> On 9/15/22 5:21 PM, Ole Holm Nielsen wrote: >> >> On 15-09-2022 16:08, Hermann Schwärzler wrote: >> >> Just out of curiosity: how do you insert the output of seff into the >> out-file of a job? >> >> Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf: >> MailProg=/usr/bin/smail >> >> Maybe I am missing something but from what I can tell smail sends an email >> and does *not* change or append to the .out file of a job... >> >> Regards, >> Hermann
smime.p7s
Description: S/MIME cryptographic signature
