I think the information was scattered across a few posts, but the union of which is correct:
- it depends on the benchmark - yes, L1/L2/L3 cache sizes can have a huge effect. I.e., once the buffer size gets bigger than the cache size, it takes more time to get the message from main RAM --> check the output from hwloc's "lstopo" tool to find your cache sizes - the specific flavor of shared memory used also has a huge effect. The default is copy-in/copy-out, but other shared memory mechanisms are also available (e.g., Linux CMA, Linux KNEM, XPMEM) Does that help? > On Mar 10, 2016, at 12:25 PM, BRADLEY, PETER C PW > <peter.c.brad...@pw.utc.com> wrote: > > This is an academic exercise, obviously. The curve shown comes from one pair > of ranks running on the same node alternating between MPI_Send and MPI_Recv. > The most likely suspect is a cache effect, but rather than assuming, I was > curious if there might be any other aspects of the implementation at work. > > Pete > > > > Pete, > > how did you measure the bandwidth ? > iirc, IMB benchmark does not reuse send and recv buffers, so the results > could be different. > also, you might want to use a logarithmic scale for the message size, so > information for small messages is easier to read. > > Cheers, > > Gilles > > On Thursday, March 10, 2016, BRADLEY, PETER C PW > <peter.c.bradley_at_[hidden]> > wrote: > > > I’m curious what causes the hump in the pingpong bandwidth curve when > > running on shared memory. Here’s an example running on a fairly > > antiquated > > single-socket 4 core laptop with linux (2.6.32 kernel). Is this a cache > > effect? Something in OpenMPI itself, or a combination? > > > > > > > > > > > > [image: Macintosh HD:Users:up:Pictures:bandwidth_onepair_onenode.png] > > > > > > > > Pete > > > > > > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28674.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
