Hi, when I try the same with mpirun, I get this:
-------------------------------------------------------------------------- An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. This could be caused by a number of factors, including an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements). ————————————————————————————————————— I use CentOS 7’s Openmpi: openmpi-1.10.3-3.el7.x86_64 There is a bug on libfabric as well, so when I run it with srun, I get: cluster.13046hfi_wait_for_device: The /dev/hfi1_0 device failed to appear after 15.0 seconds: Connection timed out Hello world! I am process number: 0 on host cluster-master cluster-slave1.9925hfi_wait_for_device: The /dev/hfi1_0 device failed to appear after 15.0 seconds: Connection timed out Hello world! I am process number: 0 on host cluster-slave1 2017-01-23 16:10 GMT+01:00 TO_Webmaster <[email protected]>: > > Is this OpenMPI? We experienced similar behaviour with OpenMPI. This > was fixed after recompiling OpenMPI with PMI, i.e. > > ./configure [...] --with-pmi=/path/to/slurm [...] > > 2017-01-23 14:22 GMT+01:00 liu junjun <[email protected]>: > > Hi Paddy, > > > > Thanks a lot for you kind helps! > > > > Replacing mpirun by srun seems still not working. Here's how I did: > >>cat a.sh > > #!/bin/bash > > srun ./mpitest > >>sbatch -n 2 ./a.sh > > Submitted batch job 3611 > >>cat slurm-3611.out > > Hello world: rank 0 of 1 running on cdd001 > > Hello world: rank 0 of 1 running on cdd001 > > > > So, srun just executed the program twice, instead of running it as > parallel. > > > > If I replace the srun back to mpirun, the good output is produced: > >>cat a.sh > > #!/bin/bash > > mpirun ./mpitest > >> sbatch a.sh > > Submitted batch job 3612 > >>cat slurm-3612.out > > Hello world: rank 0 of 2 running on cdd001 > > Hello world: rank 1 of 2 running on cdd001 > > > > I also tried using srun inside bash script for serial program: > >>cat a.sh > > #!/bin/bash > > srun echo Hello > >>sbatch -n 2 ./a.sh > > Submitted batch job 3614 > >>cat slurm-3614.out > > Hello > > Hello > > > > Any idea? > > > > Thanks in advance! > > > > Junjun > > > > > > > > On Mon, Jan 23, 2017 at 6:16 PM, Paddy Doyle <[email protected]> wrote: > >> > >> > >> Hi Junjun, > >> > >> On Mon, Jan 23, 2017 at 12:04:17AM -0800, liu junjun wrote: > >> > >> > Hi all, > >> > > >> > I have small MPI test program just printing the rannk id of a parallel > >> > job. > >> > The output is like this: > >> > >mpirun -n 2 ./mpitest > >> > Hello world: rank 0 of 2 running on cddlogin > >> > Hello world: rank 1 of 2 running on cddlogin > >> > > >> > I ran this test program with salloc. It produces similar output: > >> > >salloc -n 2 > >> > salloc: Granted job allocation 3605 > >> > >mpirun -n 2 ./mpitest > >> > Hello world: rank 0 of 2 running on cdd001 > >> > Hello world: rank 1 of 2 running on cdd001 > >> > > >> > I put this one line command into a bash script for running with > sbatch. > >> > It > >> > also get the same result as expected. However, it is totally different > >> > if > >> > it run with srun: > >> > >srun -n 2 mpirun -n 2 ./mpitest > >> > Hello world: rank 0 of 2 running on cdd001 > >> > Hello world: rank 1 of 2 running on cdd001 > >> > Hello world: rank 0 of 2 running on cdd001 > >> > Hello world: rank 1 of 2 running on cdd001 > >> > >> That looks like expected behaviour from calling both srun and mpirun; > have > >> never > >> tried it, but it looks like what might happen if you call them both. > >> > >> But it's not recommended to run your code like that. > >> > >> I think basically don't call both srun and mpirun! In your sbatch either > >> put: > >> > >> #SBATCH -n 2 > >> .... > >> mpirun ./mpitest > >> > >> > >> ..or: > >> > >> > >> #SBATCH -n 2 > >> .... > >> srun ./mpitest > >> > >> > >> You don't need both. And it's simpler not to repeat the '-n 2' again in > >> the > >> mpirun/srun line, as it will lead to copy/paste errors when you change > it > >> in the > >> '#SBATCH' line but not below. > >> > >> > The test program was invoked twice ($SLURM_NTASKS) with each time > asked > >> > 2 > >> > ($SLURM_NTASKS) CPU for mpi program!! > >> > >> Yes. > >> > >> > The problem of srun is actually not about mpi: > >> > >srun -n 2 echo "Hello" > >> > Hello > >> > Hello > >> > > >> > How can I resolve the problem of srun, and let it behaves like sbatch > or > >> > salloc, where the program executed only one time? > >> > > >> > The version of slurm is 16.05.3, and > >> > >> Thanks, > >> Paddy > >> > >> -- > >> Paddy Doyle > >> Trinity Centre for High Performance Computing, > >> Lloyd Building, Trinity College Dublin, Dublin 2, Ireland. > >> Phone: +353-1-896-3725 > >> http://www.tchpc.tcd.ie/ > > > > > -- [] Alexandre Strube [email protected]
