Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-17 Thread Charles A Taylor
Just to follow up… This turned out to be a bug in OpenMPI+UCX. https://github.com/openucx/ucx/issues/2921 https://github.com/open-mpi/ompi/pull/5878 I cherry-picked the patch from the github master

Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-06 Thread Charles A Taylor
> On Oct 6, 2018, at 6:06 AM, wrote: > > Charles, > > ucx has a higher priority than ob1, that is why it is used by default > when available. Good to know. Thanks. > > If you can provide simple instructions on how to build and test one of > the apps that experiment > a memory leak, that

Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-06 Thread gilles
Charles, ucx has a higher priority than ob1, that is why it is used by default when available. If you can provide simple instructions on how to build and test one of the apps that experiment a memory leak, that would greatly help us and the UCX folks reproduce, troubleshoot and diagnose this

Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-06 Thread Charles A Taylor
> On Oct 5, 2018, at 11:31 AM, Gilles Gouaillardet > wrote: > > are you saying that even if you > > mpirun --mca pml ob1 ... > > (e.g. force the ob1 component of the pml framework) the memory leak is > still present ? No, I do not mean to say that - at least not in the current incarnatio

Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-05 Thread Gilles Gouaillardet
Charles, are you saying that even if you mpirun --mca pml ob1 ... (e.g. force the ob1 component of the pml framework) the memory leak is still present ? As a side note, we strongly recommend to avoid configure --with-FOO=/usr instead configure --with-FOO should be used (otherwise you will end u

Re: [OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-05 Thread Pavel Shamis
Posting this on UCX list. On Thu, Oct 4, 2018 at 4:42 PM Charles A Taylor wrote: > > We are seeing a gaping memory leak when running OpenMPI 3.1.x (or 2.1.2, > for that matter) built with UCX support. The leak shows up > whether the “ucx” PML is specified for the run or not. The applications

[OMPI users] Memory Leak in 3.1.2 + UCX

2018-10-04 Thread Charles A Taylor
We are seeing a gaping memory leak when running OpenMPI 3.1.x (or 2.1.2, for that matter) built with UCX support. The leak shows up whether the “ucx” PML is specified for the run or not. The applications in question are arepo and gizmo but it I have no reason to believe that others are not af