Hello, Chris. The results doesn't seem valid and relevant to your problem for the following reasons: a. lines "rank: 9437456 key:PMI_netinfo_of_task val:" are definitely contain bad rank so either PMI init is buggy or you run that test incorrectly (second is more likely). b. I don't see the line "mpi/pmi2: failed to send temp kvs to compute nodes" that was the reason of this topic.
Looking at the command line: sbatch -p debug --wrap "srun -vvvvvv --slurmd-debug=verbose ./testpmi2" I don't see --mpi=pmi2 so that explains everything :). Can you run it once again as follows: *sbatch -p debug --wrap "srun --mpi=pmi2 -vvvvvv --slurmd-debug=verbose ./testpmi2"* 2015-09-04 4:15 GMT+03:00 Christopher Samuel <[email protected]>: > On 02/09/15 16:48, Artem Polyakov wrote: > > > Hi, Chris. > > Could you rerun with > > > > *srun -vvvvvv --slurmd-debug=verbose* > > > > and provide the output? > > I've re-run with the testpmi2 binary that can be built (thanks David), > the output file is 10MB so I've gzip'd it. > > The command line to generate this was: > > [samuel@snowy-m PMI2]$ sbatch -p debug --wrap "srun -vvvvvv > --slurmd-debug=verbose ./testpmi2" > Submitted batch job 996 > > All the best, > Chris > -- > Christopher Samuel Senior Systems Administrator > VLSCI - Victorian Life Sciences Computation Initiative > Email: [email protected] Phone: +61 (0)3 903 55545 > http://www.vlsci.org.au/ http://twitter.com/vlsci > > -- С Уважением, Поляков Артем Юрьевич Best regards, Artem Y. Polyakov
