/********** The problem ***********/ I have a cluster with 10GE ethernet and 100Gb infiniband. While running my application - CAMx, I found that the performance with IB is not as good as ethernet. That is confusing because IB latency and bandwith is undoubtablely better than ethernet, which is proven by MPI benchmark IMB-MPI1 and osu.
/********** software stack ***********/ centos7.4 with kernel 4.11.0-45.6.1.el7a.aarch64 MLNX_OFED_LINUX-4.3-1.0.1.0 from http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers gnu7.3 from OpenHPC release. yun install gnu7-compilers-ohpc-7.3.0-43.1.aarch64 openmpi3 from OpenHPC release. yum install openmpi3-gnu7-ohpc-3.0.0-36.4.aarch64 CAMx 6.4.0 from http://www.camx.com/ IMB from https://github.com/intel/mpi-benchmarks OSU from http://mvapich.cse.ohio-state.edu/benchmarks/ /********** command lines are ********/ (time mpirun --allow-run-as-root -mca btl self,openib -x OMP_NUM_THREADS=2 -n 32 -mca btl_tcp_if_include eth2 ../../src/CAMx.v6.40.openMPI.gfortranomp.ompi) > camx_openib_log 2>&1 (time mpirun --allow-run-as-root -mca btl self,tcp -x OMP_NUM_THREADS=2 -n 32 -mca btl_tcp_if_include eth2 ../../src/CAMx.v6.40.openMPI.gfortranomp.ompi) > camx_tcp_log 2>&1 (time mpirun --allow-run-as-root -mca btl self,openib -x OMP_NUM_THREADS=2 -n 32 -mca btl_tcp_if_include eth2 IMB-MPI1 allreduce -msglog 8 -npmin 1000) > IMB_openib_log 2>&1 (time mpirun --allow-run-as-root -mca btl self,tcp -x OMP_NUM_THREADS=2 -n 32 -mca btl_tcp_if_include eth2 IMB-MPI1 allreduce -msglog 8 -npmin 1000) > IMB_tcp_log 2>&1 (time mpirun --allow-run-as-root -mca btl self,openib -x OMP_NUM_THREADS=2 -n 32 -mca btl_tcp_if_include eth2 osu_latency) > osu_openib_log 2>&1 (time mpirun --allow-run-as-root -mca btl self,tcp -x OMP_NUM_THREADS=2 -n 32 -mca btl_tcp_if_include eth2 osu_latency) > osu_tcp_log 2>&1 /********** about openmpi and network config *************/ Please refer to relevant log files in the attachment. *Best Regards,* *Xie Bin*
ompi_support.tar.bz2
Description: application/bzip2
_______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
