Thanks, that works. I also opened a UCX bug report and I already got a fix for it:
https://github.com/openucx/ucx/issues/4224 With that patch also UCX detects user namespace correctly. Adrian On Tue, Sep 24, 2019 at 12:12:54PM -0600, Nathan Hjelm wrote: > You can use the uct btl if you want to come to use UCX but want Vader for > shared memory. Typical usage is --mca pml ob1 --mca osc ^ucx --mca btl > self,vader,uct --mca byl_uct_memory_domains ib/mlx5_0 > > -Nathan > > > On Sep 24, 2019, at 11:13 AM, Adrian Reber via users > > <users@lists.open-mpi.org> wrote: > > > > Now that my PR to autodetect user namespaces has been merged in Open MPI > > (thanks everyone for the help!) I tried running containers on UCX > > enabled installation. The whole UCX setup confuses me a lot. > > > > Is it possible with UCX enabled installation to tell Open MPI to use > > vader for shared memory and not UCX? Because UCX seems to have similar > > assumptions for its shared memory communication as vader, that processes > > can talk to each other somehow: > > > > mm_posix.c:445 UCX ERROR Error returned from open in attach. Permission > > denied. File name is: /proc/24149/fd/16 > > mm_ep.c:75 UCX ERROR failed to connect to remote peer with mm. remote > > mm_id: 103719165231238 > > pml_ucx.c:383 Error: ucp_ep_create(proc=6) failed: Shared memory error > > > > If I disable UCX '--mca pml ^ucx', shared memory communication works again, > > but network based communication is not happening for also unknown reasons. > > > > I tried to configure UCX with environment variables based on > > https://github.com/openucx/ucx/wiki/UCX-environment-parameters > > but that did not work. > > > > So my question is, how can I use Open MPI with UCX, but vader for > > local communication? > > > > Everything I am doing uses user namespace based containers for every > > process. > > > > Adrian > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org > > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
