Thanks for the help!!!! I renamed the nodes, and now slurm and openmpi seem
to be playing nicely with each other.
Bob
On 1/19/07, Jeff Squyres <jsquy...@cisco.com> wrote:
I think the SLURM code in Open MPI is making an assumption that is
failing in your case: we assume that your nodes will have a specific
naming convention:
mycluster.example.com --> head node
mycluster01.example.com --> cluster node 1
mycluster02.example.com --> cluster node 2
...etc.
OMPI is therefore parsing the SLURM environment and not correctly
groking the "master,wolf1" string because, to be honest, I didn't
even know that SLURM supported that scenario. I.e., I thought SLURM
required the naming convention I listed above. In hindsight, that's
a pretty silly assumption, but to be fair, you're the first user that
ever came to us with this problem (i.e., we use pretty much the same
string parsing in LAM/MPI, which has had SLURM support for several
years). Oops!
We can fix this, but I don't know if it'll make the v1.2 cutoff or
not. :-\
Thanks for bringing this to our attention!
On Jan 19, 2007, at 1:50 PM, Robert Bicknell wrote:
> Thanks for your response. The program that I have been using for
> testing purposes is a simple hello:
>
>
> #include < stdio.h>
>
> #include <mpi.h>
>
>
> #include <sys/time.h>
> #include <sys/resource.h>
> #include <unistd.h>
> #include <stdio.h>
> main(int argc, char *argv)
> {
> char name[BUFSIZ];
> int length;
> int rank;
> struct rlimit rlim;
> FILE *output;
>
> MPI_Init(&argc, &argv);
> MPI_Get_processor_name(name, &length);
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
> rank = 0;
> MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>
> // while(1) {
> printf("%s: hello world from rank %d\n", name, rank);
> sleep(1);
> // }
> MPI_Finalize();
> }
>
> If I run this program not in a slurm environment I get the following
>
> mpirun -np 4 -mca btl tcp,self -host wolf1,master ./hello
>
> master: hello world from rank 1
> wolf1: hello world from rank 0
> wolf1: hello world from rank 2
> master: hello world from rank 3
>
> This is exactly what I expect. Now if I create a slurm environment
> using the following:
>
> srun -n 4 -A
>
> The output of printenv|grep SLRUM gives me:
>
> SLURM_NODELIST=master,wolf1
> SLURM_SRUN_COMM_PORT=58929
> SLURM_MEM_BIND_TYPE=
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_MEM_BIND_LIST=
> SLURM_CPU_BIND_LIST=
> SLURM_NNODES=2
> SLURM_JOBID=66135
> SLURM_TASKS_PER_NODE=2(x2)
> SLURM_SRUN_COMM_HOST=master
> SLURM_CPU_BIND_TYPE=
> SLURM_MEM_BIND_VERBOSE=quiet
> SLURM_NPROCS=4
>
> This seems to indicate that both master and wolf1 have been
> allocated and that each node should run 2 tasks, which is correct
> since both master and wolf1 are dual processor machines.
>
> Now if I run:
>
> mpirun -np 4 -mca btl tcp,self ./hello
>
> The output is:
>
> master: hello world from rank 1
> master: hello world from rank 2
> master: hello world from rank 3
> master: hello world from rank 0
>
>
> All four processes are running on master and none on wolf1.
>
> If I try the following and specify the hosts. I get the following
> error message.
>
> mpirun -np 4 -host wolf1,master -mca btl tcp,self ./hello
>
> ----------------------------------------------------------------------
> ----
> Some of the requested hosts are not included in the current
> allocation for the
> application:
> ./hello
> The requested hosts were:
> wolf1,master
>
> Verify that you have mapped the allocated resources properly using the
> --host specification.
> ----------------------------------------------------------------------
> ----
> [master:28022] [0,0,0] ORTE_ERROR_LOG: Out of resource in file
> rmgr_urm.c at line 377
> [master:28022] mpirun: spawn failed with errno=-2
>
>
> I'm at a loss to figure out how to get this working correctly. Any
> help would be greatly appreciated.
>
> Bob
>
> On 1/19/07, Ralph Castain < r...@lanl.gov> wrote: Open MPI and SLURM
> should work together just fine right out-of-the-box. The
> typical command progression is:
>
> srun -n x -A
> mpirun -n y .....
>
>
> If you are doing those commands and still see everything running on
> the head
> node, then two things could be happening:
>
> (a) you really aren't getting an allocation from slurm. Perhaps you
> don't
> have slurm setup correctly and aren't actually seeing the
> allocation in your
> environment. Do a "printenv | grep SLURM" and see if you find the
> following
> variables:
> SLURM_NPROCS=8
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=
> SLURM_CPU_BIND_LIST=
> SLURM_MEM_BIND_VERBOSE=quiet
> SLURM_MEM_BIND_TYPE=
> SLURM_MEM_BIND_LIST=
> SLURM_JOBID=47225
> SLURM_NNODES=2
> SLURM_NODELIST=odin[013-014]
> SLURM_TASKS_PER_NODE=4(x2)
> SLURM_SRUN_COMM_PORT=43206
> SLURM_SRUN_COMM_HOST=odin
>
> Obviously, the values will be different, but we really need the
> TASKS_PER_NODE and NODELIST ones to be there
>
> (b) the master node is being included in your nodelist and you aren't
> running enough mpi processes to need more nodes (i.e., the number
> of slots
> on the master node is greater than or equal to the num procs you
> requested).
> You can force Open MPI to not run on your master node by including
> "--nolocal" on your command line.
>
> Of course, if the master node is the only thing on the nodelist,
> this will
> cause mpirun to abort as there is nothing else for us to use.
>
> Hope that helps
> Ralph
>
>
> On 1/18/07 11:03 PM, "Robert Bicknell" <robbickn...@gmail.com> wrote:
>
> > I'm trying to get slurm and openmpi to work together on a debian,
> two
> > node cluster. Slurm and openmpi seem to work fine seperately,
> but when
> > I try to run a mpi program in a slurm allocation, all the
> processes get
> > run on the master node, and not distributed to the second node.
> What am
> > I doing wrong?
> >
> > Bob
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
