On 04/09/15 15:44, Christopher Samuel wrote: > I'll test that again once the compute nodes are back online again.
OK, back to the right error again. [samuel@snowy-m PMI2]$ srun -p debug --mpi=pmi2 ./testpmi2 srun: job 1004 queued and waiting for resources srun: job 1004 has been allocated resources srun: error: mpi/pmi2: failed to send temp kvs to compute nodes slurmstepd: *** STEP 1004.0 CANCELLED AT 2015-09-04T15:53:38 *** on snowy010 srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: error: snowy010: task 0: Killed I've attached the output file for the version with debugging on, and I have a suspicion it's related to: srun: debug: slurm_forward_data: nodelist=snowy010, address=/tmp/sock.pmi2.1006.0, len=243 We are using the tmpdir spank plugin to map /tmp for a job to a temporary directory created on the scratch filesystem (rather than having it hit the tiny ramdisk on our diskless nodes - we've been unable to get a number of codes to honour $TMPDIR, etc). I can disable that plugin to check that theory. I wonder if there's a way to tell Slurm and PMI2 to use /dev/shm instead for this? cheers, Chris -- Christopher Samuel Senior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: [email protected] Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci
slurm-1006.out.gz
Description: GNU Zip compressed data
