Bill, thankyou very much for that. I guess I have to get my systemd hat on. A hat which is very large and composed of many parts, and indeed functions as a pair of pants too.
On 10 August 2017 at 14:33, Bill Barth <[email protected]> wrote: > If you use a modern enough OS (RHEL/CentOS 7, etc), XDG_RUNTIME_DIR will > probably be set and mounted (it’s a tmpfs with a limited max size mounted, > per-session, under /run/user/<uid>) on your login nodes, any node that > environment propagates to (like the first compute node of a job), and > anywhere that the user (or MPI stack) sshes to due to the PAM integration > of pam_systemd.so in the auth process. Just having the environment variable > set is not quite enough, though, you also need it mounted and unmounted at > the end of each shell session. If you add the same line from > /etc/pam.d/system-auth (or your OS’s equivalent) to /etc/pam.d/slurm, then > srun- and sbatch-initiated shells and processes will also have the > directory properly set up. MPI jobs that use ssh will get the mount > automatically due to the ssh PAM integration with systemd, but those that > use PMI-* and srun need the additional PAM integration. > > Like it or not, this systemd-based/freedesktop.org system for a private, > ephemeral temporary directory appears to be the future on Linux, and lots > of GUI-based programs (Qt) are already expecting it. There are instructions > in the standard for what you’re supposed to do as a developer if it doesn’t > exist or has the wrong permissions, but this method is at least becoming > standardized across Linux distributions. We first discovered this recently > on some new CentOS 7 boxes that we were running under SLURM and were > complaining in some GUI apps that didn’t have it mounted. It took a little > while to figure out where in the PAM stack to insert the pam_systemd.so > configuration line to guarantee that it was working for all our SLURM jobs, > but the above method seems to solve the problem. > > Best, > Bill. > > -- > Bill Barth, Ph.D., Director, HPC > [email protected] | Phone: (512) 232-7069 > Office: ROC 1.435 | Fax: (512) 475-9445 > > > > On 8/10/17, 3:06 AM, "Fokke Dijkstra" <[email protected]> wrote: > > We use the spank-private-tmp plugin developed at HPC2N in Sweden: > > https://github.com/hpc2n/spank-private-tmp > > > > See also: https://slurm.schedmd.com/SUG14/private_tmp.pdf > for a presentation about the plugin. > > > > > 2017-08-10 9:31 GMT+02:00 John Hearns <[email protected]>: > > I am sure someone discussed this topic on this list a few months > ago... if it rings any bells please let me know. > I am not discussing setting the TMPDIR environment variable and > crateing a new TMPDIR directory on a per job basis - though thankyou for > the help I did get when discussing this. > > > Rather I would like to set up a new namespace when a job runs such > that /tmp is unique to every job. /tmp can of course be a directly > uniquely created for that job and deleted afterwards. > This is to cope with any software which writes to /tmp rather than > using the TMPDIR variable. > If anyone does this let me know what your techniques are please. > > > > > > > > > > > > -- > Fokke Dijkstra > <[email protected]> <mailto:[email protected]> > Research and Innovation Support > Center for Information Technology, University of Groningen > Postbus 11044, 9700 CA Groningen, The Netherlands > +31-50-363 9243 > > > > > > > >
