[OMPI users] Slurm and Openmpi
I'm trying to get slurm and openmpi to work together on a debian, two node cluster. Slurm and openmpi seem to work fine seperately, but when I try to run a mpi program in a slurm allocation, all the processes get run on the master node, and not distributed to the second node. What am I doing wrong? Bob
Re: [OMPI users] Slurm and Openmpi
Thanks for your response. The program that I have been using for testing
purposes is a simple hello:
#include
#include
#include
#include
#include
#include
main(int argc, char *argv)
{
char name[BUFSIZ];
int length;
int rank;
struct rlimit rlim;
FILE *output;
MPI_Init(&argc, &argv);
MPI_Get_processor_name(name, &length);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
rank = 0;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
// while(1) {
printf("%s: hello world from rank %d\n", name, rank);
sleep(1);
// }
MPI_Finalize();
}
If I run this program not in a slurm environment I get the following
mpirun -np 4 -mca btl tcp,self -host wolf1,master ./hello
master: hello world from rank 1
wolf1: hello world from rank 0
wolf1: hello world from rank 2
master: hello world from rank 3
This is exactly what I expect. Now if I create a slurm environment using
the following:
srun -n 4 -A
The output of printenv|grep SLRUM gives me:
SLURM_NODELIST=master,wolf1
SLURM_SRUN_COMM_PORT=58929
SLURM_MEM_BIND_TYPE=
SLURM_CPU_BIND_VERBOSE=quiet
SLURM_MEM_BIND_LIST=
SLURM_CPU_BIND_LIST=
SLURM_NNODES=2
SLURM_JOBID=66135
SLURM_TASKS_PER_NODE=2(x2)
SLURM_SRUN_COMM_HOST=master
SLURM_CPU_BIND_TYPE=
SLURM_MEM_BIND_VERBOSE=quiet
SLURM_NPROCS=4
This seems to indicate that both master and wolf1 have been allocated and
that each node should run 2 tasks, which is correct since both master and
wolf1 are dual processor machines.
Now if I run:
mpirun -np 4 -mca btl tcp,self ./hello
The output is:
master: hello world from rank 1
master: hello world from rank 2
master: hello world from rank 3
master: hello world from rank 0
All four processes are running on master and none on wolf1.
If I try the following and specify the hosts. I get the following error
message.
mpirun -np 4 -host wolf1,master -mca btl tcp,self ./hello
--
Some of the requested hosts are not included in the current allocation for
the
application:
./hello
The requested hosts were:
wolf1,master
Verify that you have mapped the allocated resources properly using the
--host specification.
--
[master:28022] [0,0,0] ORTE_ERROR_LOG: Out of resource in file rmgr_urm.c at
line 377
[master:28022] mpirun: spawn failed with errno=-2
I'm at a loss to figure out how to get this working correctly. Any help
would be greatly appreciated.
Bob
On 1/19/07, Ralph Castain wrote:
Open MPI and SLURM should work together just fine right out-of-the-box.
The
typical command progression is:
srun -n x -A
mpirun -n y .
If you are doing those commands and still see everything running on the
head
node, then two things could be happening:
(a) you really aren't getting an allocation from slurm. Perhaps you don't
have slurm setup correctly and aren't actually seeing the allocation in
your
environment. Do a "printenv | grep SLURM" and see if you find the
following
variables:
SLURM_NPROCS=8
SLURM_CPU_BIND_VERBOSE=quiet
SLURM_CPU_BIND_TYPE=
SLURM_CPU_BIND_LIST=
SLURM_MEM_BIND_VERBOSE=quiet
SLURM_MEM_BIND_TYPE=
SLURM_MEM_BIND_LIST=
SLURM_JOBID=47225
SLURM_NNODES=2
SLURM_NODELIST=odin[013-014]
SLURM_TASKS_PER_NODE=4(x2)
SLURM_SRUN_COMM_PORT=43206
SLURM_SRUN_COMM_HOST=odin
Obviously, the values will be different, but we really need the
TASKS_PER_NODE and NODELIST ones to be there
(b) the master node is being included in your nodelist and you aren't
running enough mpi processes to need more nodes (i.e., the number of slots
on the master node is greater than or equal to the num procs you
requested).
You can force Open MPI to not run on your master node by including
"--nolocal" on your command line.
Of course, if the master node is the only thing on the nodelist, this will
cause mpirun to abort as there is nothing else for us to use.
Hope that helps
Ralph
On 1/18/07 11:03 PM, "Robert Bicknell" wrote:
> I'm trying to get slurm and openmpi to work together on a debian, two
> node cluster. Slurm and openmpi seem to work fine seperately, but when
> I try to run a mpi program in a slurm allocation, all the processes get
> run on the master node, and not distributed to the second node. What am
> I doing wrong?
>
> Bob
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Slurm and Openmpi
Thanks for the help I renamed the nodes, and now slurm and openmpi seem
to be playing nicely with each other.
Bob
On 1/19/07, Jeff Squyres wrote:
I think the SLURM code in Open MPI is making an assumption that is
failing in your case: we assume that your nodes will have a specific
naming convention:
mycluster.example.com --> head node
mycluster01.example.com --> cluster node 1
mycluster02.example.com --> cluster node 2
...etc.
OMPI is therefore parsing the SLURM environment and not correctly
groking the "master,wolf1" string because, to be honest, I didn't
even know that SLURM supported that scenario. I.e., I thought SLURM
required the naming convention I listed above. In hindsight, that's
a pretty silly assumption, but to be fair, you're the first user that
ever came to us with this problem (i.e., we use pretty much the same
string parsing in LAM/MPI, which has had SLURM support for several
years). Oops!
We can fix this, but I don't know if it'll make the v1.2 cutoff or
not. :-\
Thanks for bringing this to our attention!
On Jan 19, 2007, at 1:50 PM, Robert Bicknell wrote:
> Thanks for your response. The program that I have been using for
> testing purposes is a simple hello:
>
>
> #include < stdio.h>
>
> #include
>
>
> #include
> #include
> #include
> #include
> main(int argc, char *argv)
> {
> char name[BUFSIZ];
> int length;
> int rank;
> struct rlimit rlim;
> FILE *output;
>
> MPI_Init(&argc, &argv);
> MPI_Get_processor_name(name, &length);
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
> rank = 0;
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>
> // while(1) {
> printf("%s: hello world from rank %d\n", name, rank);
> sleep(1);
> // }
> MPI_Finalize();
> }
>
> If I run this program not in a slurm environment I get the following
>
> mpirun -np 4 -mca btl tcp,self -host wolf1,master ./hello
>
> master: hello world from rank 1
> wolf1: hello world from rank 0
> wolf1: hello world from rank 2
> master: hello world from rank 3
>
> This is exactly what I expect. Now if I create a slurm environment
> using the following:
>
> srun -n 4 -A
>
> The output of printenv|grep SLRUM gives me:
>
> SLURM_NODELIST=master,wolf1
> SLURM_SRUN_COMM_PORT=58929
> SLURM_MEM_BIND_TYPE=
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_MEM_BIND_LIST=
> SLURM_CPU_BIND_LIST=
> SLURM_NNODES=2
> SLURM_JOBID=66135
> SLURM_TASKS_PER_NODE=2(x2)
> SLURM_SRUN_COMM_HOST=master
> SLURM_CPU_BIND_TYPE=
> SLURM_MEM_BIND_VERBOSE=quiet
> SLURM_NPROCS=4
>
> This seems to indicate that both master and wolf1 have been
> allocated and that each node should run 2 tasks, which is correct
> since both master and wolf1 are dual processor machines.
>
> Now if I run:
>
> mpirun -np 4 -mca btl tcp,self ./hello
>
> The output is:
>
> master: hello world from rank 1
> master: hello world from rank 2
> master: hello world from rank 3
> master: hello world from rank 0
>
>
> All four processes are running on master and none on wolf1.
>
> If I try the following and specify the hosts. I get the following
> error message.
>
> mpirun -np 4 -host wolf1,master -mca btl tcp,self ./hello
>
> --
>
> Some of the requested hosts are not included in the current
> allocation for the
> application:
> ./hello
> The requested hosts were:
> wolf1,master
>
> Verify that you have mapped the allocated resources properly using the
> --host specification.
> --
>
> [master:28022] [0,0,0] ORTE_ERROR_LOG: Out of resource in file
> rmgr_urm.c at line 377
> [master:28022] mpirun: spawn failed with errno=-2
>
>
> I'm at a loss to figure out how to get this working correctly. Any
> help would be greatly appreciated.
>
> Bob
>
> On 1/19/07, Ralph Castain < r...@lanl.gov> wrote: Open MPI and SLURM
> should work together just fine right out-of-the-box. The
> typical command progression is:
>
> srun -n x -A
> mpirun -n y .
>
>
> If you are doing those commands and still see everything running on
> the head
> node, then two things could be happening:
>
> (a) you really aren't getting an allocation from slurm. Perhaps you
> don't
> have slurm setup correctly and aren't actually seeing the
> allocation in your
> environment. Do a "printenv | grep SLURM" and see if you find the
> following
> variables:
> SLURM_NPROCS=8
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=
> SLURM_CPU_BIND_LIST=
> SLURM_MEM_
