Peter, mpirun does fork&exec the orted daemon on the remote nodes, an then orted firm&exec the MPI app (your shell script here) the MPI app connects back to orted (1.6.5 might use tcp instead of unix sockets, but I do not recall if 127.0.0.1 is hardcoded, and if it is, what the impact of having the MPI app in a container is) the environment used by the MPI app starts with OMPI_, so you might want to update your she'll script first.
I wrote a high level overview of these mechanisms a few days ago at http://www.open-mpi.org/community/lists/devel/2015/10/18189.php, feel free to read it. at first glance, the way you are using containers look overly complicated. are you sure you want one MPI task per container ? I mean, you could have one container per VM and several MPI tasks per container. then, a simpler alternative could be to wrap mpirun, starts the containers and then mpirun "into" these containers. Cheers, Gilles On Sunday, October 18, 2015, Peter Davis <peter.dav...@gmail.com> wrote: > Hello, > > I am currently trying to get MPI working using docker containers across > networked virtual machines. The problem I have is that I don't seem to get > any output from a dockerized MPI program (a tutorial matrix multiply > program), but inspecting the nodes (docker containers) each process sits at > 98% cpu usage whilst mpirun does not seem to get any output, and nothing > complains or finishes. Running the same application on my laptop (with a > local mpirun call) the running time is less than a second. > > Running a “hello world” program which just printed its “rank" seems to > work fine with this docker setup, with mpirun getting the stdout printf > calls. However, that example never had any inter-node communications. > > Much of the system setup we have is based on the idea that environment > variables are what passes information from mpirun to the node processes > (under ssh), but I have not had any exposure to MPI prior to this, so > perhaps this is wrong. > > A (relatively) quick description of the system; > > - We have 3 virtual machines that are interconnected on an infiniband > network (however, in the hope of it being simpler, we currently use the > tcp/ip layer not infiniband) > > - There is no resource manager (e.g. Slurm) running, everything is over > ssh. > > - Each vm is running Centos 7, has openmpi 1.6.4 installed, and loads the > mpi environment module up via .bashrc. > > - We have docker installed on each vm. > > - We have a container that is based on ubuntu 14:04, has openmpi version > 1.6.5 installed, and runs an mpi-based matrix_multiply program on startup. > Ompi_info output is: > > > ompi_info -v ompi full --parsable > > package:Open MPI buildd@allspice Distribution > > ompi:version:full:1.6.5 > > ompi:version:svn:r28673 > > ompi:version:release_date:Jun 26, 2013 > > orte:version:full:1.6.5 > > orte:version:svn:r28673 > > orte:version:release_date:Jun 26, 2013 > > opal:version:full:1.6.5 > > opal:version:svn:r28673 > > opal:version:release_date:Jun 26, 2013 > > mpi-api:version:full:2.1 > > ident:1.6.5 > > > > - To start the mpi program(s), on one of the vms we call mpirun with: > > mpirun --mca btl_tcp_port_min_v4 7950 --mca btl_tcp_port_range_v4 50 > --mca btl tcp,self --mca mpi_base_verbose 30 -H > 10.3.2.41,10.3.2.42,10.3.2.43 /root/docker-machine-script.sh > > - This then runs our shell script on the vm, which collects environment > variables, filters out the vm mpi variables, and starts the docker > container with those env variables. Our shell script is: > > #!/bin/bash > > env | grep MPI | grep -v > 'MPI_INCLUDE\|MPI_PYTHON\|MPI_LIB\|MPI_BIN\|MPI_COMPILER\|MPI_SYSCONFIG\|MPI_SUFFIX\|MPI_MAN\|MPI_HOME\|MPI_FORTRAN_MOD_DIR' > > /root/env.txt > > docker run --privileged --env-file /root/env.txt -p 7950-8000:7950-8000 > mrmagooey/mpi-mm > > > - Finally, the container runs the mpi application (the matrix multiply) > using the environment variables. > > - Each container is capable of ssh’ing into the other containers without > password. > > It is a rather complicated setup, but the real application that we will > eventually run is a pain to compile from scratch and so docker is a very > appealing solution. > > Attached is the "ompi_info —all” and “ifconfig” called on one of the host > vm’s (as per the forum support instructions). Also attached is the result > of /usr/bin/printenv on one of the containers. > > Thank you, > > Peter > > >
