I have most of the code ready, but I still have troubles doing
OPAL_MODEX_RECV. I am using the following lines, based on the code from
orte/test/mpi/pmix.c:
OPAL_MODEX_SEND_VALUE(rc, OPAL_PMIX_LOCAL, "user_ns_id", &value, OPAL_INT);
This sets rc to 0. For receiving:
OPAL_MODEX_RECV_VALUE(rc, "user_ns_id", &wildcard_rank, &ptr, OPAL_INT);
and rc is always set to -13. Is this how it is supposed to work, or do I
have to do it differently?
Adrian
On Mon, Jul 22, 2019 at 02:03:20PM +0000, Ralph Castain via users wrote:
> If that works, then it might be possible to include the namespace ID in the
> job-info provided by PMIx at startup - would have to investigate, so please
> confirm that the modex option works first.
>
> > On Jul 22, 2019, at 1:22 AM, Gilles Gouaillardet via users
> > <users@lists.open-mpi.org> wrote:
> >
> > Adrian,
> >
> >
> > An option is to involve the modex.
> >
> > each task would OPAL_MODEX_SEND() its own namespace ID, and then
> > OPAL_MODEX_RECV()
> >
> > the one from its peers and decide whether CMA support can be enabled.
> >
> >
> > Cheers,
> >
> >
> > Gilles
> >
> > On 7/22/2019 4:53 PM, Adrian Reber via users wrote:
> >> I had a look at it and not sure if it really makes sense.
> >>
> >> In btl_vader_{put,get}.c it would be easy to check for the user
> >> namespace ID of the other process, but the function would then just
> >> return OPAL_ERROR a bit earlier instead of as a result of
> >> process_vm_{read,write}v(). Nothing would really change.
> >>
> >> A better place for the check would be mca_btl_vader_check_single_copy()
> >> but I do not know if at this point the PID of the other processes is
> >> already known. Not sure if I can check for the user namespace ID of the
> >> other processes.
> >>
> >> Any recommendations how to do this?
> >>
> >> Adrian
> >>
> >> On Sun, Jul 21, 2019 at 03:08:01PM -0400, Nathan Hjelm wrote:
> >>> Patches are always welcome. What would be great is a nice big warning
> >>> that CMA support is disabled because the processes are on different
> >>> namespaces. Ideally all MPI processes should be on the same namespace to
> >>> ensure the best performance.
> >>>
> >>> -Nathan
> >>>
> >>>> On Jul 21, 2019, at 2:53 PM, Adrian Reber via users
> >>>> <users@lists.open-mpi.org> wrote:
> >>>>
> >>>> For completeness I am mentioning my results also here.
> >>>>
> >>>> To be able to mount file systems in the container it can only work if
> >>>> user namespaces are used and even if the user IDs are all the same (in
> >>>> each container and on the host), to be able to ptrace the kernel also
> >>>> checks if the processes are in the same user namespace (in addition to
> >>>> being owned by the same user). This check - same user namespace - fails
> >>>> and so process_vm_readv() and process_vm_writev() will also fail.
> >>>>
> >>>> So Open MPI's checks are currently not enough to detect if 'cma' can be
> >>>> used. Checking for the same user namespace would also be necessary.
> >>>>
> >>>> Is this a use case important enough to accept a patch for it?
> >>>>
> >>>> Adrian
> >>>>
> >>>>> On Fri, Jul 12, 2019 at 03:42:15PM +0200, Adrian Reber via users wrote:
> >>>>> Gilles,
> >>>>>
> >>>>> thanks again. Adding '--mca btl_vader_single_copy_mechanism none' helps
> >>>>> indeed.
> >>>>>
> >>>>> The default seems to be 'cma' and that seems to use process_vm_readv()
> >>>>> and process_vm_writev(). That seems to require CAP_SYS_PTRACE, but
> >>>>> telling Podman to give the process CAP_SYS_PTRACE with
> >>>>> '--cap-add=SYS_PTRACE'
> >>>>> does not seem to be enough. Not sure yet if this related to the fact
> >>>>> that Podman is running rootless. I will continue to investigate, but now
> >>>>> I know where to look. Thanks!
> >>>>>
> >>>>> Adrian
> >>>>>
> >>>>>> On Fri, Jul 12, 2019 at 06:48:59PM +0900, Gilles Gouaillardet via
> >>>>>> users wrote:
> >>>>>> Adrian,
> >>>>>>
> >>>>>> Can you try
> >>>>>> mpirun --mca btl_vader_copy_mechanism none ...
> >>>>>>
> >>>>>> Please double check the MCA parameter name, I am AFK
> >>>>>>
> >>>>>> IIRC, the default copy mechanism used by vader directly accesses the
> >>>>>> remote process address space, and this requires some permission
> >>>>>> (ptrace?) that might be dropped by podman.
> >>>>>>
> >>>>>> Note Open MPI might not detect both MPI tasks run on the same node
> >>>>>> because of podman.
> >>>>>> If you use UCX, then btl/vader is not used at all (pml/ucx is used
> >>>>>> instead)
> >>>>>>
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Gilles
> >>>>>>
> >>>>>> Sent from my iPod
> >>>>>>
> >>>>>>> On Jul 12, 2019, at 18:33, Adrian Reber via users
> >>>>>>> <users@lists.open-mpi.org> wrote:
> >>>>>>>
> >>>>>>> So upstream Podman was really fast and merged a PR which makes my
> >>>>>>> wrapper unnecessary:
> >>>>>>>
> >>>>>>> Add support for --env-host :
> >>>>>>> https://github.com/containers/libpod/pull/3557
> >>>>>>>
> >>>>>>> As commented in the PR I can now start mpirun with Podman without a
> >>>>>>> wrapper:
> >>>>>>>
> >>>>>>> $ mpirun --hostfile ~/hosts --mca orte_tmpdir_base /tmp/podman-mpirun
> >>>>>>> podman run --env-host --security-opt label=disable -v
> >>>>>>> /tmp/podman-mpirun:/tmp/podman-mpirun --userns=keep-id --net=host
> >>>>>>> mpi-test /home/mpi/ring
> >>>>>>> Rank 0 has cleared MPI_Init
> >>>>>>> Rank 1 has cleared MPI_Init
> >>>>>>> Rank 0 has completed ring
> >>>>>>> Rank 0 has completed MPI_Barrier
> >>>>>>> Rank 1 has completed ring
> >>>>>>> Rank 1 has completed MPI_Barrier
> >>>>>>>
> >>>>>>> This is example was using TCP and on an InfiniBand based system I have
> >>>>>>> to map the InfiniBand devices into the container.
> >>>>>>>
> >>>>>>> $ mpirun --mca btl ^openib --hostfile ~/hosts --mca orte_tmpdir_base
> >>>>>>> /tmp/podman-mpirun podman run --env-host -v
> >>>>>>> /tmp/podman-mpirun:/tmp/podman-mpirun --security-opt label=disable
> >>>>>>> --userns=keep-id --device /dev/infiniband/uverbs0 --device
> >>>>>>> /dev/infiniband/umad0 --device /dev/infiniband/rdma_cm --net=host
> >>>>>>> mpi-test /home/mpi/ring
> >>>>>>> Rank 0 has cleared MPI_Init
> >>>>>>> Rank 1 has cleared MPI_Init
> >>>>>>> Rank 0 has completed ring
> >>>>>>> Rank 0 has completed MPI_Barrier
> >>>>>>> Rank 1 has completed ring
> >>>>>>> Rank 1 has completed MPI_Barrier
> >>>>>>>
> >>>>>>> This is all running without root and only using Podman's rootless
> >>>>>>> support.
> >>>>>>>
> >>>>>>> Running multiple processes on one system, however, still gives me an
> >>>>>>> error. If I disable vader I guess that Open MPI is using TCP for
> >>>>>>> localhost communication and that works. But with vader it fails.
> >>>>>>>
> >>>>>>> The first error message I get is a segfault:
> >>>>>>>
> >>>>>>> [test1:00001] *** Process received signal ***
> >>>>>>> [test1:00001] Signal: Segmentation fault (11)
> >>>>>>> [test1:00001] Signal code: Address not mapped (1)
> >>>>>>> [test1:00001] Failing at address: 0x7fb7b1552010
> >>>>>>> [test1:00001] [ 0] /lib64/libpthread.so.0(+0x12d80)[0x7f6299456d80]
> >>>>>>> [test1:00001] [ 1]
> >>>>>>> /usr/lib64/openmpi/lib/openmpi/mca_btl_vader.so(mca_btl_vader_send+0x3db)[0x7f628b33ab0b]
> >>>>>>> [test1:00001] [ 2]
> >>>>>>> /usr/lib64/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send_request_start_rdma+0x1fb)[0x7f62901d24bb]
> >>>>>>> [test1:00001] [ 3]
> >>>>>>> /usr/lib64/openmpi/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_send+0xfd6)[0x7f62901be086]
> >>>>>>> [test1:00001] [ 4]
> >>>>>>> /usr/lib64/openmpi/lib/libmpi.so.40(PMPI_Send+0x1bd)[0x7f62996f862d]
> >>>>>>> [test1:00001] [ 5] /home/mpi/ring[0x400b76]
> >>>>>>> [test1:00001] [ 6]
> >>>>>>> /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f62990a3813]
> >>>>>>> [test1:00001] [ 7] /home/mpi/ring[0x4008be]
> >>>>>>> [test1:00001] *** End of error message ***
> >>>>>>>
> >>>>>>> Guessing that vader uses shared memory this is expected to fail, with
> >>>>>>> all the namespace isolations in place. Maybe not with a segfault, but
> >>>>>>> each container has its own shared memory. So next step was to use the
> >>>>>>> host's ipc and pid namespace and mount /dev/shm:
> >>>>>>>
> >>>>>>> '-v /dev/shm:/dev/shm --ipc=host --pid=host'
> >>>>>>>
> >>>>>>> Which does not segfault, but still does not look correct:
> >>>>>>>
> >>>>>>> Rank 0 has cleared MPI_Init
> >>>>>>> Rank 1 has cleared MPI_Init
> >>>>>>> Rank 2 has cleared MPI_Init
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> [test1:17722] Read -1, expected 80000, errno = 1
> >>>>>>> Rank 0 has completed ring
> >>>>>>> Rank 2 has completed ring
> >>>>>>> Rank 0 has completed MPI_Barrier
> >>>>>>> Rank 1 has completed ring
> >>>>>>> Rank 2 has completed MPI_Barrier
> >>>>>>> Rank 1 has completed MPI_Barrier
> >>>>>>>
> >>>>>>> This is using the Open MPI ring.c example with SIZE increased from 20
> >>>>>>> to 20000.
> >>>>>>>
> >>>>>>> Any recommendations what vader needs to communicate correctly?
> >>>>>>>
> >>>>>>> Adrian
> >>>>>>>
> >>>>>>>> On Thu, Jul 11, 2019 at 12:07:35PM +0200, Adrian Reber via users
> >>>>>>>> wrote:
> >>>>>>>> Gilles,
> >>>>>>>>
> >>>>>>>> thanks for pointing out the environment variables. I quickly created
> >>>>>>>> a
> >>>>>>>> wrapper which tells Podman to re-export all OMPI_ and PMIX_ variables
> >>>>>>>> (grep "\(PMIX\|OMPI\)"). Now it works:
> >>>>>>>>
> >>>>>>>> $ mpirun --hostfile ~/hosts ./wrapper -v /tmp:/tmp --userns=keep-id
> >>>>>>>> --net=host mpi-test /home/mpi/hello
> >>>>>>>>
> >>>>>>>> Hello, world (2 procs total)
> >>>>>>>> --> Process # 0 of 2 is alive. ->test1
> >>>>>>>> --> Process # 1 of 2 is alive. ->test2
> >>>>>>>>
> >>>>>>>> I need to tell Podman to mount /tmp from the host into the
> >>>>>>>> container, as
> >>>>>>>> I am running rootless I also need to tell Podman to use the same
> >>>>>>>> user ID
> >>>>>>>> in the container as outside (so that the Open MPI files in /tmp) can
> >>>>>>>> be
> >>>>>>>> shared and I am also running without a network namespace.
> >>>>>>>>
> >>>>>>>> So this is now with the full Podman provided isolation except the
> >>>>>>>> network namespace. Thanks for you help!
> >>>>>>>>
> >>>>>>>> Adrian
> >>>>>>>>
> >>>>>>>>> On Thu, Jul 11, 2019 at 04:47:21PM +0900, Gilles Gouaillardet via
> >>>>>>>>> users wrote:
> >>>>>>>>> Adrian,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> the MPI application relies on some environment variables (they
> >>>>>>>>> typically
> >>>>>>>>> start with OMPI_ and PMIX_).
> >>>>>>>>>
> >>>>>>>>> The MPI application internally uses a PMIx client that must be able
> >>>>>>>>> to
> >>>>>>>>> contact a PMIx server
> >>>>>>>>>
> >>>>>>>>> (that is included in mpirun and the orted daemon(s) spawned on the
> >>>>>>>>> remote
> >>>>>>>>> hosts).
> >>>>>>>>>
> >>>>>>>>> located on the same host.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> If podman provides some isolation between the app inside the
> >>>>>>>>> container (e.g.
> >>>>>>>>> /home/mpi/hello)
> >>>>>>>>>
> >>>>>>>>> and the outside world (e.g. mpirun/orted), that won't be an easy
> >>>>>>>>> ride.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Gilles
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 7/11/2019 4:35 PM, Adrian Reber via users wrote:
> >>>>>>>>>> I did a quick test to see if I can use Podman in combination with
> >>>>>>>>>> Open
> >>>>>>>>>> MPI:
> >>>>>>>>>>
> >>>>>>>>>> [test@test1 ~]$ mpirun --hostfile ~/hosts podman run
> >>>>>>>>>> quay.io/adrianreber/mpi-test /home/mpi/hello
> >>>>>>>>>>
> >>>>>>>>>> Hello, world (1 procs total)
> >>>>>>>>>> --> Process # 0 of 1 is alive. ->789b8fb622ef
> >>>>>>>>>>
> >>>>>>>>>> Hello, world (1 procs total)
> >>>>>>>>>> --> Process # 0 of 1 is alive. ->749eb4e1c01a
> >>>>>>>>>>
> >>>>>>>>>> The test program (hello) is taken from
> >>>>>>>>>> https://raw.githubusercontent.com/openhpc/ohpc/obs/OpenHPC_1.3.8_Factory/tests/mpi/hello.c
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> The problem with this is that each process thinks it is process 0
> >>>>>>>>>> of 1
> >>>>>>>>>> instead of
> >>>>>>>>>>
> >>>>>>>>>> Hello, world (2 procs total)
> >>>>>>>>>> --> Process # 1 of 2 is alive. ->test1
> >>>>>>>>>> --> Process # 0 of 2 is alive. ->test2
> >>>>>>>>>>
> >>>>>>>>>> My questions is how is the rank determined? What resources do I
> >>>>>>>>>> need to have
> >>>>>>>>>> in my container to correctly determine the rank.
> >>>>>>>>>>
> >>>>>>>>>> This is Podman 1.4.2 and Open MPI 4.0.1.
> >>>>>>>>>>
> >>>>>>>>>> Adrian
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> users mailing list
> >>>>>>>>>> users@lists.open-mpi.org
> >>>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users
> >>>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> users mailing list
> >>>>>>>>> users@lists.open-mpi.org
> >>>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users
> >>>>>>>> _______________________________________________
> >>>>>>>> users mailing list
> >>>>>>>> users@lists.open-mpi.org
> >>>>>>>> https://lists.open-mpi.org/mailman/listinfo/users
> >>>>>>> _______________________________________________
> >>>>>>> users mailing list
> >>>>>>> users@lists.open-mpi.org
> >>>>>>> https://lists.open-mpi.org/mailman/listinfo/users
> >>>>>> _______________________________________________
> >>>>>> users mailing list
> >>>>>> users@lists.open-mpi.org
> >>>>>> https://lists.open-mpi.org/mailman/listinfo/users
> >>>>> _______________________________________________
> >>>>> users mailing list
> >>>>> users@lists.open-mpi.org
> >>>>> https://lists.open-mpi.org/mailman/listinfo/users
> >>>> _______________________________________________
> >>>> users mailing list
> >>>> users@lists.open-mpi.org
> >>>> https://lists.open-mpi.org/mailman/listinfo/users
> >> Adrian
> >>
> > _______________________________________________
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
>
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
