My problem: grant licensed software availability to my users only if
they request it on slurm; for now with local licenses.
I wrote a job_submit lua script which checks job_desc.licenses and if
it contains the appropriate strings it sets an appropriate
SOMETHING_LICENSE_REQ environmental variable.
This part works, I can see the environmental variable correctly set in
the jobs that require the license.
Now this licensed software is a bit tricky to manage, so the way that
I thought to use it is simply to make its binary disappear from the
nodes when not requested, with a prolog and epilog scripts which copy
it from a location in the shared filesystem accessible only by root.
Simply a copy during the prolog and a delete during the epilog.
Something like this:
if [[ $SOMETHING_LICENSE_REQ == 1 ]]; then
# copy the binary
fi
After banging my head to make the prolog run (executable bit and full
path required, and not said so in the documentation and error logs
being cryptic about it) I am finally able to see it running.... only
to find out that the SOMETHING_LICENSE_REQ environmental variable is
not set, despite the documentation at
https://slurm.schedmd.com/prolog_epilog.html stating
> The task prolog is executed with the same environment as the user tasks to be
> initiated.
Now, I'd be very happy to do this copy from the job_submit, but that
is run on the head node (I checked) and so I can't do that. It would
seem strange that the job_submit is run after the prolog, since the
latter runs on the compute node (I checked that too).
Moreover, I also verified with additional environmental variables
which I set at submit time are available for the user job correctly,
but not in the prolog.
So either I misinterpreted that "same environment as the user tasks"
or there is something else that I am doing wrong.
Does anybody have any insight?
