Hi Chris,
I actually had the same idea concerning XFS project quotas when I came
across that private-tmp SPANK plugin recently. Haven't had the
opportunity to make a test implementation yet, though.
I was wondering if constantly making and deleting XFS projects has a
considerable impact on performance and stability. So I'd be glad if you
could share some of your experience with that setup.
Also, would you mind providing access to your prolog and epilog scripts?
Greetings from Germany,
Christoph
On 22/11/2018 00.37, Christopher Samuel wrote:
On 22/11/18 12:38 am, Douglas Duckworth wrote:
We are setting TmpFS=/scratchLocal in /etc/slurm/slurm.conf on nodes
and controller. However $TMPDIR value seems to be /tmp not
/scratchLocal. As a result users are writing to /tmp which we do not
want.
Our solution to that was to use a plugin that creates a per-job
directory and then maps it into the job as /tmp via namespaces.
The advantage of this is that it works for applications & runtimes that
don't honour ${TMPDIR} (yes, Java, I'm looking at you).
https://github.com/chrissamuel/spank-private-tmp
I've got a branch there that does the same for /dev/shm (which relies on
/dev/shm/slurm existing).
Because our local scratch space is using XFS our prolog sets an XFS
project quota using the job ID to be the requested space via --tmp (and
our submit filter converts --tmp requests into --gres:tmp:${TMP}M
requests so they are used for scheduling so we don't overcommit space).
Our epilog then cleans both the per job temporary space and per job
/dev/shm up at the end.
All the best,
Chris
--
Dr. Christoph Brüning
Universität Würzburg
Rechenzentrum
Am Hubland
D-97074 Würzburg
Tel.: +49 931 31-80499
