I'm afraid I'm confused by your question, so please clarify: the hostnames that have ethernet interfaces do NOT have infiniband interfaces on them? If so, why would you expect OMPI to use infiniband when you execute on those hosts? Or do they contain an infiniband interface in addition to the ethernet interface?
On Mar 14, 2010, at 4:47 PM, nad labassi wrote: > Hi, > > My problem is: I installed openmpi 1.2.9, with mvapi support, but the > execution is done on ethernet, unless I use ipoib. > > I built openmpi with the command: > > ./configure CC=/home/pgi/linux86-64/6.2/bin/pgcc > CXX=/home/pgi/linux86-64/6.2/bin/pgCC > FC=/home/pgi/linux86-64/6.2/bin/pgf90 > --without-tm --without-gm --without-mx --with-sge --without-openib > --with-mvapi=/usr/mellanox --with-mvapi-libdir=/usr/lib64 > --with-threads=posix > --prefix=/home/openmpi-1.2.9_mvapi > > I run the program with the commande: > > /home/openmpi-1.2.9_mvapi/bin/mpirun --mca btl mvapi,self -n 8 > -machinefile /home/myaccount/machines > /home/myaccount/mpihello_openmpi-1.2.9.exe > > When the machines file contains hostnames for ethernet interfaces, > the execution is never done on infiniband > > When the machines file contains hostnames for ib interfaces (ipoib), > then the execution is done on infiniband. > > I want the execution to be done on infiniband, while the machines > file contains hostnames for ethernet interfaces. > > > ompi_info --param btl mvapi outputs are: > > MCA btl: parameter "btl_base_debug" (current value: > "0") > If btl_base_debug is 1 standard debug is > output, if > 1 verbose debug is output > MCA btl: parameter "btl" (current value: <none>) > Default selection set of components for the > btl framework (<none> means "use all > components that can be found") > MCA btl: parameter "btl_base_verbose" (current > value: "0") > Verbosity level for the btl framework (0 = > no verbosity) > MCA btl: parameter "btl_mvapi_max_btls" (current > value: "4") > maximum number of HCAs/ports to use > MCA btl: parameter "btl_mvapi_free_list_num" > (current value: "8") > initial size of free lists > MCA btl: parameter "btl_mvapi_free_list_max" > (current value: "-1") > maximum size of free lists > MCA btl: parameter "btl_mvapi_free_list_inc" > (current value: "32") > increment size of free lists > MCA btl: parameter "btl_mvapi_mpool" (current value: > "rdma") > name of the memory pool to be used > MCA btl: parameter "btl_mvapi_reg_mru_len" (current > value: "16") > length of the registration cache most > recently used list > MCA btl: parameter "btl_mvapi_use_srq" (current > value: "0") > if 1 use the IB shared receive queue to > post receive descriptors > MCA btl: parameter "btl_mvapi_ib_cq_size" (current > value: "10000") > size of the IB completion queue > MCA btl: parameter "btl_mvapi_ib_sg_list_size" > (current value: "1") > size of IB segment list > MCA btl: parameter "btl_mvapi_ib_pkey_ix" (current > value: "0") > IB pkey index > MCA btl: parameter "btl_mvapi_ib_psn" (current > value: "0") > IB Packet sequence starting number > MCA btl: parameter "btl_mvapi_ib_qp_ous_rd_atom" > (current value: "4") > IB outstanding atomic reads > > MCA btl: parameter "btl_mvapi_ib_mtu" (current > value: "3") > IB MTU, in bytes. Valid values are: 1=256 > bytes, 2=512 bytes, 3=1024 bytes, 4=2 > 048 bytes, 5=4096 bytes. > MCA btl: parameter "btl_mvapi_ib_min_rnr_timer" > (current value: "5") > IB min rnr timer > MCA btl: parameter "btl_mvapi_ib_timeout" (current > value: "10") > IB transmit timeout > MCA btl: parameter "btl_mvapi_ib_retry_count" > (current value: "7") > IB transmit retry count > MCA btl: parameter "btl_mvapi_ib_rnr_retry" (current > value: "7") > IB rnr retry > MCA btl: parameter "btl_mvapi_ib_max_rdma_dst_ops" > (current value: "4") > IB max rdma destination operations > MCA btl: parameter "btl_mvapi_ib_service_level" > (current value: "0") > IB service level > MCA btl: parameter "btl_mvapi_ib_static_rate" > (current value: "0") > (current value: "0") > IB static rate > MCA btl: parameter "btl_mvapi_ib_src_path_bits" > (current value: "0") > IB source path bits > MCA btl: parameter "btl_mvapi_rd_num" (current > value: "8") > number of receive descriptors to post to a > QP > MCA btl: parameter "btl_mvapi_rd_low" (current > value: "6") > low water mark before reposting occurs > MCA btl: parameter "btl_mvapi_rd_win" (current > value: "4") > window size at which generate explicity > credit message > MCA btl: parameter "btl_mvapi_srq_rd_max" (current > value: "1000") > Maximum number of receive descriptors > posted per SRQ. > > MCA btl: parameter "btl_mvapi_srq_rd_per_peer" > (current value: "16") > receive descriptors posted per peer, SRQ > mode only > MCA btl: parameter "btl_mvapi_srq_sd_max" (current > value: "8") > Maximum number of send descriptors posted > per process > MCA btl: parameter "btl_mvapi_use_eager_rdma" > (current value: "1") > user RDMA for eager messages > MCA btl: parameter "btl_mvapi_eager_rdma_threshold" > (current value: "16") > Open rdma channel for eager messages after > this number of messages received from > peer (zero to disable) > MCA btl: parameter "btl_mvapi_max_eager_rdma" > > (current value: "16") > Maximum number of eager RDMA connections > MCA btl: parameter "btl_mvapi_eager_rdma_num" > (current value: "16") > Number of RDMA buffers for eager messages > MCA btl: parameter "btl_mvapi_exclusivity" (current > value: "1024") > BTL exclusivity > MCA btl: parameter "btl_mvapi_eager_limit" (current > value: "12288") > eager send limit > MCA btl: parameter "btl_mvapi_min_send_size" > (current value: "32768") > minimum send size > MCA btl: parameter "btl_mvapi_max_send_size" > (current value: "65536") > maximum send size > MCA btl: parameter "btl_mvapi_min_rdma_size" > (current value: "1048576") > minimum rdma size > MCA btl: parameter "btl_mvapi_max_rdma_size" > (current value: "1048576") > maximium rdma size > MCA btl: parameter "btl_mvapi_flags" (current value: > "6") > BTL flags, SEND=1, PUT=2, GET=4 > MCA btl: parameter "btl_mvapi_bandwidth" (current > value: "800") > Approximate maximum bandwidth of > interconnect > MCA btl: parameter "btl_mvapi_priority" (current > value: "0") > MCA btl: parameter "btl_base_warn_component_unused" > (current value: "1") > This parameter is used to turn on warning > messages when certain NICs are not use > > > Thanks a lot, > > Nadjet. > > > > > Hotmail : une messagerie fiable avec une protection anti-spam performante > Inscrivez-vous_______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
