Re: [OMPI users] openib/mpi_alloc_mem pathology

2017-03-16 Thread Alfio Lazzaro
Hello Dave and others, we jump in the discussion as CP2K developers. We would like to ask you which version of CP2K you are using in your tests and if you can share with us your input file and output log. Some clarifications on the way we use MPI allocate/free: 1) only buffers used for MPI communi

Re: [OMPI users] openib/mpi_alloc_mem pathology [#20160912-1315]

2017-03-16 Thread Paul Kapinos
Hi, On 03/16/17 10:35, Alfio Lazzaro wrote: We would like to ask you which version of CP2K you are using in your tests Release 4.1 and if you can share with us your input file and output log. The question goes to Mr Mathias Schumacher, on CC: Best Paul Kapinos (Our internal ticketing sys

[OMPI users] Suppressing Nvidia warnings

2017-03-16 Thread Roland Fehrenbacher
Hi, OpenMPI 2.0.2 built with cuda support brings up lots of warnings like NVIDIA: no NVIDIA devices found when running on HW without Nvidia devices. Is there a way to suppress these warnings? It would be quite a hassle to maintain different OpenMPI builds on clusters with just some GPU machines.

Re: [OMPI users] openib/mpi_alloc_mem pathology

2017-03-16 Thread Jingchao Zhang
Hi Jeff, One of my earlier replies includes the backtraces of cp2k.popt process and the problem points to MPI_ALLOC_MEM/MPI_FREE_MEM. https://mail-archive.com/users@lists.open-mpi.org/msg30587.html If that part of the code is commented out, is there another way for openmpi to find that backt

Re: [OMPI users] openib/mpi_alloc_mem pathology

2017-03-16 Thread Jeff Squyres (jsquyres)
On Mar 16, 2017, at 10:37 AM, Jingchao Zhang wrote: > > One of my earlier replies includes the backtraces of cp2k.popt process and > the problem points to MPI_ALLOC_MEM/MPI_FREE_MEM. > https://mail-archive.com/users@lists.open-mpi.org/msg30587.html Yep -- saw it. That -- paired with the profil

Re: [OMPI users] openib/mpi_alloc_mem pathology

2017-03-16 Thread Jingchao Zhang
I meant "another way to find a backtrace with MPI_ALLOC_MEM/MPI_FREE_MEM being ancestors of a write() system call on /dev/infiniband/verbs, i.e., doing RDMA over IB." I read your opened issue #3183 and I think we are on the right track. Yay~ Cheers, Jingchao ___

Re: [OMPI users] Suppressing Nvidia warnings

2017-03-16 Thread Sylvain Jeaugey
Hi Roland, I can't find this message in the Open MPI source code. Could it be hwloc ? Some other library you are using ? Sylvain On 03/16/2017 04:23 AM, Roland Fehrenbacher wrote: Hi, OpenMPI 2.0.2 built with cuda support brings up lots of warnings like NVIDIA: no NVIDIA devices found whe

Re: [OMPI users] openib/mpi_alloc_mem pathology

2017-03-16 Thread Paul Kapinos
Jeff, I confirm: your patch did it. (tried on 1.10.6 - do not even need to rebuild the cp2k.popt , just load another Open MPI version compiled with Jeff'path) ( On Intel OmpiPath the same speed as with --mca btl ^tcp,openib ) On 03/16/17 01:03, Jeff Squyres (jsquyres) wrote: It looks like t

Re: [OMPI users] openib/mpi_alloc_mem pathology

2017-03-16 Thread Jeff Squyres (jsquyres)
Ok. I talked with Nathan about this a bit. Here's what we think we should do: 1. Add an MCA param to disable (de)registration as part of ALLOC/FREE_MEM. Because that's just the Open MPI way (moar MCA paramz!). 2. If memory hooks are enabled, default to not (de)registering as part of ALLOC/FR

Re: [OMPI users] Suppressing Nvidia warnings

2017-03-16 Thread Roland Fehrenbacher
> "SJ" == Sylvain Jeaugey writes: Hi Sylvain, SJ> Hi Roland, I can't find this message in the Open MPI source SJ> code. Could it be hwloc ? Some other library you are using ? the message comes from libnvidia-ml.so.x.y which libmpi is linked against. Thanks, Roland SJ> On 03/1