Jeff et al,
Thanks, exactly what I was looking for.
Pete
I think the information was scattered across a few posts, but the union of
which is correct:
- it depends on the benchmark
- yes, L1/L2/L3 cache sizes can have a huge effect. I.e., once the buffer size
gets bigger than the cache size, it takes more time to get the message from
main RAM
--> check the output from hwloc's "lstopo" tool to find your cache sizes
- the specific flavor of shared memory used also has a huge effect. The default
is copy-in/copy-out, but other shared memory mechanisms are also available
(e.g., Linux CMA, Linux KNEM, XPMEM)
Does that help?
> On Mar 10, 2016, at 12:25 PM, BRADLEY, PETER C PW
> <peter.c.bradley_at_[hidden]> wrote:
>
> This is an academic exercise, obviously. The curve shown comes from one pair
> of ranks running on the same node alternating between MPI_Send and MPI_Recv.
> The most likely suspect is a cache effect, but rather than assuming, I was
> curious if there might be any other aspects of the implementation at work.
>
> Pete
>
>
>
> Pete,
>
> how did you measure the bandwidth ?
> iirc, IMB benchmark does not reuse send and recv buffers, so the results
> could be different.
> also, you might want to use a logarithmic scale for the message size, so
> information for small messages is easier to read.
>
> Cheers,
>
> Gilles
>
> On Thursday, March 10, 2016, BRADLEY, PETER C PW <peter.c.bradley_at_[hidden]>
> wrote:
>
> > I’m curious what causes the hump in the pingpong bandwidth curve when
> > running on shared memory. Here’s an example running on a fairly antiquated
> > single-socket 4 core laptop with linux (2.6.32 kernel). Is this a cache
> > effect? Something in OpenMPI itself, or a combination?
> >
> >
> >
> >
> >
> > [image: Macintosh HD:Users:up:Pictures:bandwidth_onepair_onenode.png]
> >
> >
> >
> > Pete
> >
> >
> >
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/03/28674.php
--
Jeff Squyres
jsquyres_at_[hidden]
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
