Hi, my system have mellanox connectX adapter in all nodes, when use cmd "/sbin/lspci", will print "0c:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR] (rev a0)". And ofed 1.4.2 has been installed. I install openmpi 1.4.3 using cmd "./configure --prefix=/home_soft/home/scksz/mpi/ompi143; make all install", the config.log contain those information about openib, xrc or xoob:
<<<<<<<<config.log information about openib<<<<<<<< configure:5726: checking whether to add padding to the openib control header configure:5739: result: no | #define OMPI_OPENIB_PAD_HDR 0 ... configure:126382: checking for m4 configure components in framework btl configure:126384: result: elan, gm, mx, ofud, openib, portals, tcp, udapl configure:132447: checking --with-openib value configure:132450: result: simple ok (unspecified) configure:132492: checking --with-openib-libdir value configure:132495: result: simple ok (unspecified) configure:135339: checking for MCA component btl:openib compile mode configure:135345: result: dso configure:135446: checking --with-openib value configure:135449: result: simple ok (unspecified) configure:135491: checking --with-openib-libdir value configure:135494: result: simple ok (unspecified) configure:138177: checking which openib btl cpcs will be built configure:138179: result: oob xoob rdmacm configure:138242: checking if MCA component btl:openib can compile configure:138244: result: yes config.status:2748: creating ompi/mca/btl/openib/Makefile MCA_btl_ALL_COMPONENTS=' self sm elan gm mx ofud openib portals tcp udapl' MCA_btl_ALL_SUBDIRS=' mca/btl/self mca/btl/sm mca/btl/elan mca/btl/gm mca/btl/mx mca/btl/ofud mca/btl/openib mca/btl/portals mca/btl/tcp mca/btl/udapl' MCA_btl_DSO_COMPONENTS=' self sm ofud openib tcp' MCA_btl_DSO_SUBDIRS=' mca/btl/self mca/btl/sm mca/btl/ofud mca/btl/openib mca/btl/tcp' MCA_btl_openib_have_ibcm_FALSE='' MCA_btl_openib_have_ibcm_TRUE='#' MCA_btl_openib_have_rdmacm_FALSE='#' MCA_btl_openib_have_rdmacm_TRUE='' MCA_btl_openib_have_xrc_FALSE='#' MCA_btl_openib_have_xrc_TRUE='' OMPI_BUILD_btl_openib_DSO_FALSE='#' OMPI_BUILD_btl_openib_DSO_TRUE='' btl_openib_CFLAGS='' btl_openib_CPPFLAGS='' btl_openib_LDFLAGS='' btl_openib_LIBS='-lrdmacm -libverbs ' <<<<<<<<config.log information about xrc<<<<<<<< configure:134262: checking for ibv_create_xrc_rcv_qp configure:134318: gcc -o conftest -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -pthread conftest.c -lnsl -lutil -lm -libverbs >&5 configure:134325: $? = 0 configure:134347: result: yes configure:134948: checking if ConnectX XRC support is enabled configure:134956: result: yes configure:137261: checking for ibv_create_xrc_rcv_qp configure:137346: result: yes configure:137947: checking if ConnectX XRC support is enabled configure:137955: result: yes | #define HAVE_IBV_CREATE_XRC_RCV_QP 1 ... | #define OMPI_HAVE_CONNECTX_XRC 1 ... ac_cv_func_ibv_create_xrc_rcv_qp=yes MCA_btl_openib_have_xrc_FALSE='#' MCA_btl_openib_have_xrc_TRUE='' <<<<<<<<config.log information about xoob<<<<<<<< configure:138179: result: oob xoob rdmacm <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< after ompi's installation I add the ompi's PATH and LD_LIBRARY_PATH in .bashrc. Then I compile a mpi code named "mpisample.c", use cmd "mpicc mpisample.c -o a_user.out". When I run the programm "run a_user.out" using cmd "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl_openib_cpc_include xoob --mca orte_base_help_aggregate 0 a_user.out", will print: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: CB060107 Local device: mlx4_0 Local port: 1 CPCs attempted: xoob -------------------------------------------------------------------------- -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: LB270210 Local device: mlx4_0 Local port: 1 CPCs attempted: xoob -------------------------------------------------------------------------- -------------------------------------------------------------------------- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: CB060106 Local device: mlx4_0 Local port: 1 CPCs attempted: xoob -------------------------------------------------------------------------- Hello! My rank is 0. My hostname is "LB270210" Hello! My rank is 2. My hostname is "CB060107" Hello! My rank is 1. My hostname is "CB060106" <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< When I use cmd "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl openib,self,sm a_user.out" or "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm a_user.out", will print correct result, but use cmd "mpirun -host LB270210,CB060106,CB060107 -np 3 --mca btl openib a_user.out", will print error mesg. I want to know: 1) When I want to use XRC, I must have a special IB switch? 2) How can I use XRC in ompi,and on which situation the XRC feature will bring benifit? 3) If this is only way to using XRC that using "-mca btl_openib_cpc_include xoob"? 4) When using xoob, which transport and network protocol ompi will use? 5) When using Qlogic adapter, how to optimize opmi? 6) What's mean of "CPC" in ompi?
