On Fri, Mar 17, 2017 at 3:33 PM, Joshua Mora <joshua_m...@usa.net> wrote:
> Thanks for the quick reply. > This test is between 2 cores that are on different CPUs. Say data has to > traverse coherent fabric (eg. QPI,UPI, cHT). > It has to go to main memory independently of cache size. Wrong assumption ? > Depends on the usage pattern. Some benchmarks have options to clean/flush the cache before each round of tests. > Can data be evicted from cache and put into cache of second core on > different > CPU without placing it first in main memory ? > It would depend on the memory coherency protocol. Usually it gets marked as shared, and as a result it might not need to be pushed into main memory right away. > I am more thinking that there is a parameter that splits large messages in > smaller ones at 64k or 128k ? > Pipelining is not the answer to all situations. Once your messages are larger than the caches, you already built memory pressure (by getting outside the cache size) so the pipelining is bound by the memory bandwidth. > This seems (wrong assumption ?) like the kind of parameter I would need for > large messages on a NIC. Coalescing data / large MTU,... Sure, but there are hard limits imposed by the hardware, especially with regards to intranode communications. Once you saturate the memory bus, you hit a pretty hard limit. George. > > Joshua > > > > > > > > > ------ Original Message ------ > Received: 02:15 PM CDT, 03/17/2017 > From: George Bosilca <bosi...@icl.utk.edu> > To: Open MPI Users <users@lists.open-mpi.org> > > Subject: Re: [OMPI users] tuning sm/vader for large messages > > > > > > > > > > > > > > > > > > > Joshua, > > > > In shared memory the bandwidth depends on many parameters, including the > > process placement and the size of the different cache levels. In your > > particular case I guess after 128k you are outside the L2 cache (1/2 of > the > > cache in fact) and the bandwidth will drop as the data need to be flushed > > to main memory. > > > > George. > > > > > > > > On Fri, Mar 17, 2017 at 1:47 PM, Joshua Mora <joshua_m...@usa.net> > wrote: > > > > > Hello, > > > I am trying to get the max bw for shared memory communications using > > > osu_[bw,bibw,mbw_mr] benchmarks. > > > I am observing a peak at ~64k/128K msg size and then drops instead of > > > sustaining it. > > > What parameters or linux config do I need to add to default openmpi > > > settings > > > to get this improved ? > > > I am already using vader and knem. > > > > > > See below one way bandwidth with peak at 64k. > > > > > > # Size Bandwidth (MB/s) > > > 1 1.02 > > > 2 2.13 > > > 4 4.03 > > > 8 8.48 > > > 16 11.90 > > > 32 23.29 > > > 64 47.33 > > > 128 88.08 > > > 256 136.77 > > > 512 245.06 > > > 1024 263.79 > > > 2048 405.49 > > > 4096 1040.46 > > > 8192 1964.81 > > > 16384 2983.71 > > > 32768 5705.11 > > > 65536 7181.11 > > > 131072 6490.55 > > > 262144 4449.59 > > > 524288 4898.14 > > > 1048576 5324.45 > > > 2097152 5539.79 > > > 4194304 5669.76 > > > > > > Thanks, > > > Joshua > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > users mailing list > > > users@lists.open-mpi.org > > > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > > > > > > > > > > > > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org > > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > > > > > > > > > > > > > > > >
_______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
