Can you please run ibv_devinfo on one of compute nodes ? It is interesting to know what kind of IB HW you have on our cluster.However, setting: -mca btl_openib_eager_limit 65536gave a 15% improvement so OpenMPI is now down to 326 (from previous 376 seconds). Still a lot more than ScaliMPI with 214 seconds.
Pasha
