Hi,
My problem is: I installed openmpi 1.2.9, with mvapi support, but the
execution is done on ethernet, unless I use ipoib.
I built openmpi with the command:
./configure CC=/home/pgi/linux86-64/6.2/bin/pgcc
CXX=/home/pgi/linux86-64/6.2/bin/pgCC
FC=/home/pgi/linux86-64/6.2/bin/pgf90
--without-tm --without-gm --without-mx --with-sge --without-openib
--with-mvapi=/usr/mellanox --with-mvapi-libdir=/usr/lib64
--with-threads=posix
--prefix=/home/openmpi-1.2.9_mvapi
I run the program with the commande:
/home/openmpi-1.2.9_mvapi/bin/mpirun --mca btl mvapi,self -n 8
-machinefile /home/myaccount/machines
/home/myaccount/mpihello_openmpi-1.2.9.exe
When the machines file contains hostnames for ethernet interfaces,
the execution is never done on infiniband
When the machines file contains hostnames for ib interfaces (ipoib),
then the execution is done on infiniband.
I want the execution to be done on infiniband, while the machines
file contains hostnames for ethernet interfaces.
ompi_info --param btl mvapi outputs are:
MCA btl: parameter "btl_base_debug" (current value:
"0")
If btl_base_debug is 1 standard debug is
output, if > 1 verbose debug is output
MCA btl: parameter "btl" (current value: <none>)
Default selection set of components for the
btl framework (<none> means "use all
components that can be found")
MCA btl: parameter "btl_base_verbose" (current
value: "0")
Verbosity level for the btl framework (0 =
no verbosity)
MCA btl: parameter "btl_mvapi_max_btls" (current
value: "4")
maximum number of HCAs/ports to use
MCA btl: parameter "btl_mvapi_free_list_num"
(current value: "8")
initial size of free lists
MCA btl: parameter "btl_mvapi_free_list_max"
(current value: "-1")
maximum size of free lists
MCA btl: parameter "btl_mvapi_free_list_inc"
(current value: "32")
increment size of free lists
MCA btl: parameter "btl_mvapi_mpool" (current value:
"rdma")
name of the memory pool to be used
MCA btl: parameter "btl_mvapi_reg_mru_len" (current
value: "16")
length of the registration cache most
recently used list
MCA btl: parameter "btl_mvapi_use_srq" (current
value: "0")
if 1 use the IB shared receive queue to
post receive descriptors
MCA btl: parameter "btl_mvapi_ib_cq_size" (current
value: "10000")
size of the IB completion queue
MCA btl: parameter "btl_mvapi_ib_sg_list_size"
(current value: "1")
size of IB segment list
MCA btl: parameter "btl_mvapi_ib_pkey_ix" (current
value: "0")
IB pkey index
MCA btl: parameter "btl_mvapi_ib_psn" (current
value: "0")
IB Packet sequence starting number
MCA btl: parameter "btl_mvapi_ib_qp_ous_rd_atom"
(current value: "4")
IB outstanding atomic reads
MCA btl: parameter "btl_mvapi_ib_mtu" (current
value: "3")
IB MTU, in bytes. Valid values are: 1=256
bytes, 2=512 bytes, 3=1024 bytes, 4=2
048 bytes, 5=4096 bytes.
MCA btl: parameter "btl_mvapi_ib_min_rnr_timer"
(current value: "5")
IB min rnr timer
MCA btl: parameter "btl_mvapi_ib_timeout" (current
value: "10")
IB transmit timeout
MCA btl: parameter "btl_mvapi_ib_retry_count"
(current value: "7")
IB transmit retry count
MCA btl: parameter "btl_mvapi_ib_rnr_retry" (current
value: "7")
IB rnr retry
MCA btl: parameter "btl_mvapi_ib_max_rdma_dst_ops"
(current value: "4")
IB max rdma destination operations
MCA btl: parameter "btl_mvapi_ib_service_level"
(current value: "0")
IB service level
MCA btl: parameter "btl_mvapi_ib_static_rate"
(current value: "0")
(current value: "0")
IB static rate
MCA btl: parameter "btl_mvapi_ib_src_path_bits"
(current value: "0")
IB source path bits
MCA btl: parameter "btl_mvapi_rd_num" (current
value: "8")
number of receive descriptors to post to a
QP
MCA btl: parameter "btl_mvapi_rd_low" (current
value: "6")
low water mark before reposting occurs
MCA btl: parameter "btl_mvapi_rd_win" (current
value: "4")
window size at which generate explicity
credit message
MCA btl: parameter "btl_mvapi_srq_rd_max" (current
value: "1000")
Maximum number of receive descriptors
posted per SRQ.
MCA btl: parameter "btl_mvapi_srq_rd_per_peer"
(current value: "16")
receive descriptors posted per peer, SRQ
mode only
MCA btl: parameter "btl_mvapi_srq_sd_max" (current
value: "8")
Maximum number of send descriptors posted
per process
MCA btl: parameter "btl_mvapi_use_eager_rdma"
(current value: "1")
user RDMA for eager messages
MCA btl: parameter "btl_mvapi_eager_rdma_threshold"
(current value: "16")
Open rdma channel for eager messages after
this number of messages received from
peer (zero to disable)
MCA btl: parameter "btl_mvapi_max_eager_rdma"
(current value: "16")
Maximum number of eager RDMA connections
MCA btl: parameter "btl_mvapi_eager_rdma_num"
(current value: "16")
Number of RDMA buffers for eager messages
MCA btl: parameter "btl_mvapi_exclusivity" (current
value: "1024")
BTL exclusivity
MCA btl: parameter "btl_mvapi_eager_limit" (current
value: "12288")
eager send limit
MCA btl: parameter "btl_mvapi_min_send_size"
(current value: "32768")
minimum send size
MCA btl: parameter "btl_mvapi_max_send_size"
(current value: "65536")
maximum send size
MCA btl: parameter "btl_mvapi_min_rdma_size"
(current value: "1048576")
minimum rdma size
MCA btl: parameter "btl_mvapi_max_rdma_size"
(current value: "1048576")
maximium rdma size
MCA btl: parameter "btl_mvapi_flags" (current value:
"6")
BTL flags, SEND=1, PUT=2, GET=4
MCA btl: parameter "btl_mvapi_bandwidth" (current
value: "800")
Approximate maximum bandwidth of
interconnect
MCA btl: parameter "btl_mvapi_priority" (current
value: "0")
MCA btl: parameter "btl_base_warn_component_unused"
(current value: "1")
This parameter is used to turn on warning
messages when certain NICs are not use
Thanks a lot,
Nadjet.
_________________________________________________________________
Hotmail : une messagerie fiable avec une protection anti-spam performante
https://signup.live.com/signup.aspx?id=60969
