Re: [OMPI users] tuning sm/vader for large messages

2017-03-17 Thread George Bosilca
On Fri, Mar 17, 2017 at 3:33 PM, Joshua Mora wrote: > Thanks for the quick reply. > This test is between 2 cores that are on different CPUs. Say data has to > traverse coherent fabric (eg. QPI,UPI, cHT). > It has to go to main memory independently of cache size. Wrong assumption ? > Depends on t

Re: [OMPI users] tuning sm/vader for large messages

2017-03-17 Thread Joshua Mora
Thanks for the quick reply. This test is between 2 cores that are on different CPUs. Say data has to traverse coherent fabric (eg. QPI,UPI, cHT). It has to go to main memory independently of cache size. Wrong assumption ? Can data be evicted from cache and put into cache of second core on different

Re: [OMPI users] tuning sm/vader for large messages

2017-03-17 Thread George Bosilca
Joshua, In shared memory the bandwidth depends on many parameters, including the process placement and the size of the different cache levels. In your particular case I guess after 128k you are outside the L2 cache (1/2 of the cache in fact) and the bandwidth will drop as the data need to be flush

[OMPI users] tuning sm/vader for large messages

2017-03-17 Thread Joshua Mora
Hello, I am trying to get the max bw for shared memory communications using osu_[bw,bibw,mbw_mr] benchmarks. I am observing a peak at ~64k/128K msg size and then drops instead of sustaining it. What parameters or linux config do I need to add to default openmpi settings to get this improved ? I am