what is your command line and setup? (ofed version, distro)
This is what was just measured w/ fdr on haswell with v1.8.8 and mxm and UD
+ mpirun -np 2 -bind-to core -display-map -mca rmaps_base_mapping_policy
dist:span -x MXM_RDMA_PORTS=mlx5_3:1 -mca rmaps_dist_device mlx5_3:1 -x
MXM_TLS=self,shm,ud osu_latency
Data for JOB [65499,1] offset 0
======================== JOB MAP ========================
Data for node: clx-orion-001 Num slots: 28 Max slots: 0 Num procs: 1
Process OMPI jobid: [65499,1] App: 0 Process rank: 0
Data for node: clx-orion-002 Num slots: 28 Max slots: 0 Num procs: 1
Process OMPI jobid: [65499,1] App: 0 Process rank: 1
=============================================================
# OSU MPI Latency Test v4.4.1
# Size Latency (us)
0 1.18
1 1.16
2 1.19
4 1.20
8 1.19
16 1.19
32 1.21
64 1.27
and w/ ob1, openib btl:
mpirun -np 2 -bind-to core -display-map -mca rmaps_base_mapping_policy
dist:span -mca rmaps_dist_device mlx5_3:1 -mca btl_if_include mlx5_3:1
-mca pml ob1 -mca btl openib,self osu_latency
# OSU MPI Latency Test v4.4.1
# Size Latency (us)
0 1.13
1 1.17
2 1.17
4 1.17
8 1.22
16 1.23
32 1.25
64 1.28
On Tue, Sep 29, 2015 at 6:49 PM, Dave Love <d.l...@liverpool.ac.uk> wrote:
> I've just compared IB p2p latency between version 1.6.5 and 1.8.8. I'm
> surprised to find that 1.8 is rather worse, as below. Assuming that's
> not expected, are there any suggestions for debugging it?
>
> This is with FDR Mellanox, between two Sandybridge nodes on the same
> blade chassis switch. The results are similar for IMB pingpong and
> osu_latency, and reproducible. I'm running both cases the same way as
> far as I can tell (e.g. core binding with 1.6 and not turning it off
> with 1.8) just rebuilding the test against between OMPI versions.
>
> The initial osu_latency figures for 1.6 are:
>
> # OSU MPI Latency Test v5.0
> # Size Latency (us)
> 0 1.16
> 1 1.24
> 2 1.23
> 4 1.23
> 8 1.26
> 16 1.27
> 32 1.30
> 64 1.36
>
> and for 1.8:
>
> # OSU MPI Latency Test v5.0
> # Size Latency (us)
> 0 1.48
> 1 1.46
> 2 1.42
> 4 1.43
> 8 1.46
> 16 1.47
> 32 1.48
> 64 1.54
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/09/27712.php
>
--
Kind Regards,
M.
