Hi Sebastian,
Sebastian Potthoff <s.potth...@uni-muenster.de> writes:
> Hi Loris
>
> We do something similar. At the end of our script pointed to by
> EpilogSlurmctld we have
>
> Using EpilogSlurmctld only works if the slurmctld user is root (or slurm with
> root privileges), right? I opted for the normal Epilog since we wanted to
> avoid running slurm as root and I don’t have to worry
> about ownership of the output file.
Yes, good point. We should look into that.
Cheers,
Loris
> Sebastian
>
> Am 16.09.2022 um 09:09 schrieb Loris Bennett <loris.benn...@fu-berlin.de>:
>
> Hi Hermann,
>
> Sebastian Potthoff <s.potth...@uni-muenster.de> writes:
>
> Hi Hermann,
>
> I happened to read along this conversation and was just solving this issue
> today. I added this part to the epilog script to make it work:
>
> # Add job report to stdout
> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut |
> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
>
> NODELIST=($(/usr/bin/scontrol show hostnames))
>
> # Only add to StdOut file if it exists and if we are the first node
> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
> then
> echo "################################# JOB REPORT
> ##################################" >> $StdOut
> /usr/bin/seff $SLURM_JOB_ID >> $StdOut
> echo
> "###############################################################################"
> >> $StdOut
> fi
>
> We do something similar. At the end of our script pointed to by
> EpilogSlurmctld we have
>
> OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
> if [ ! -f "$OUT" ]; then
> exit
> fi
>
> printf "\n== Epilog Slurmctld
> ==================================================\n\n" >> ${OUT}
>
> seff ${SLURM_JOB_ID} >> ${OUT}
>
> printf
> "\n======================================================================\n"
> >> ${OUT}
>
> chown ${user} ${OUT}
>
> Cheers,
>
> Loris
>
> Contrary to what it says in the slurm docs
> https://slurm.schedmd.com/prolog_epilog.html I was not able to use the env
> var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I
> had to
> make sure it is only called by the „leading“ node as the epilog script will
> be called by ALL nodes of a multinode job and they would all call seff and
> clutter up the output. Last thing was to check if
> StdOut is
> not of length zero (i.e. it exists). Interactive jobs would otherwise cause
> the node to drain.
>
> Maybe this helps.
>
> Kind regards
> Sebastian
>
> PS: goslmailer looks quite nice with its recommendations! Will definitely
> look into it.
>
> --
> Westfälische Wilhelms-Universität (WWU) Münster
> WWU IT
> Sebastian Potthoff (eScience / HPC)
>
> Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler
> <hermann.schwaerz...@uibk.ac.at>:
>
> Hi Ole,
>
> On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>
> On 15-09-2022 16:08, Hermann Schwärzler wrote:
>
> Just out of curiosity: how do you insert the output of seff into the
> out-file of a job?
>
> Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf:
> MailProg=/usr/bin/smail
>
> Maybe I am missing something but from what I can tell smail sends an email
> and does *not* change or append to the .out file of a job...
>
> Regards,
> Hermann
>
--
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
