On 22/11/18 12:38 am, Douglas Duckworth wrote:
We are setting TmpFS=/scratchLocal in /etc/slurm/slurm.conf on nodes
and controller. However $TMPDIR value seems to be /tmp not
/scratchLocal. As a result users are writing to /tmp which we do not
want.
Our solution to that was to use a plugin that creates a per-job
directory and then maps it into the job as /tmp via namespaces.
The advantage of this is that it works for applications & runtimes that
don't honour ${TMPDIR} (yes, Java, I'm looking at you).
https://github.com/chrissamuel/spank-private-tmp
I've got a branch there that does the same for /dev/shm (which relies on
/dev/shm/slurm existing).
Because our local scratch space is using XFS our prolog sets an XFS
project quota using the job ID to be the requested space via --tmp (and
our submit filter converts --tmp requests into --gres:tmp:${TMP}M
requests so they are used for scheduling so we don't overcommit space).
Our epilog then cleans both the per job temporary space and per job
/dev/shm up at the end.
All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
