On 04/09/15 16:02, Christopher Samuel wrote: > I've attached the output file for the version with debugging on, and I > have a suspicion it's related to: > > srun: debug: slurm_forward_data: nodelist=snowy010, > address=/tmp/sock.pmi2.1006.0, len=243 > > We are using the tmpdir spank plugin to map /tmp for a job to a > temporary directory created on the scratch filesystem (rather than > having it hit the tiny ramdisk on our diskless nodes - we've been > unable to get a number of codes to honour $TMPDIR, etc). > > I can disable that plugin to check that theory.
...and it works.. [samuel@snowy-m PMI2]$ srun -p debug --mpi=pmi2 ./testpmi2 srun: job 1007 queued and waiting for resources srun: job 1007 has been allocated resources rank: 0 key:PMI_netinfo_of_task val:(snowy010,(eth0,IP_V4,10.14.103.10),(ib0,IP_V4,10.7.103.10),(eth0,IP_V6,fe80::42f2:e9ff:fec5:6906%eth0),(ib0,IP_V6,fe80::e61d:2 rank: 0 key:david@0 val:rbxqrwfoeloksanm rank: 0 key:mpi_reserved_ports val: 11.639000 So time to see if we can hack PMI2 to use /dev/shm instead of /tmp. -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: [email protected] Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci
