Hi all, sorry for the long long latency - this message was buried in my mailbox for months....
On 03/16/2017 10:35 AM, Alfio Lazzaro wrote: > Hello Dave and others, > we jump in the discussion as CP2K developers. > We would like to ask you which version of CP2K you are using in your tests version 4.1 (release) > and > if you can share with us your input file and output log. The input file is property of Mathias Schumacher (CC:) and we need a permission of him to provide it. > Some clarifications on the way we use MPI allocate/free: > 1) only buffers used for MPI communications are allocated with MPI > allocate/free > 2) in general we use memory pools, therefore we reach a limit in the buffers > sizes after some iterations, i.e. they are not reallocated anymore > 3) there are some cases where we don't use memory pools, but their overall > contribution should be very small. You can run with the CALLGRAPH option > (https://www.cp2k.org/dev:profiling#the_cp2k_callgraph) to get more insight > where those allocations/deallocations are. We ran the data set again with CALLGRAPH option. Please have a look at the attached files. You see a callgraph file (from rank 0 of 24 used) and some exported call tree views. We can see that the *allocate* routines (mp_[de|]allocate_[i|d]) are called 33k vs. 28k times (multiple this with 24x processes per node). In the 'good case' (Intel MPI and Open MPI with workaround) these calls are only a fraction of 1% of time; in 'bad case' (OpenMPI w/o workaround, attached) the two mp_dealocate_[i|d] calls use 81% of the time in 'Self', huh. That's mainly the observation we made a long time ago: if in a node with Intel OmniPath fabric the failback to InfiniBand is not prohibited, the MPI_Free_mem() take ages. (I'm not familiar with CCachegrind so forgive me if I'm not true). Have a nice day, Paul Kapinos -- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, IT Center Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915
20171019-callgraph.tar.gz
Description: application/gzip
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
