Hi Mike,
thanks for the help; yes, the version of MXM has 'centos5u7' in its name, I thought it to be the best match to my Centos 5.8 since there is none signed 'centos5u8' and installing 'centos6u0' fails complaining of different versions for GLIBC. The output of 'ibv_devinfo -v' for both machines is: [cut] [fsimula@agape4 ~]$ sudo /usr/bin/ibv_devinfo -v hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.7.000 node_guid: 0025:90ff:ff07:0acc sys_image_guid: 0025:90ff:ff07:0acf vendor_id: 0x02c9 vendor_part_id: 26428 hw_ver: 0xB0 board_id: SM_1061000001000 phys_port_cnt: 1 max_mr_size: 0xffffffffffffffff page_size_cap: 0xfffffe00 max_qp: 261824 max_qp_wr: 16351 device_cap_flags: 0x006c9c76 max_sge: 32 max_sge_rd: 0 max_cq: 65408 max_cqe: 4194303 max_mr: 524272 max_pd: 32764 max_qp_rd_atom: 16 max_ee_rd_atom: 0 max_res_rd_atom: 4189184 max_qp_init_rd_atom: 128 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_HCA (1) max_ee: 0 max_rdd: 0 max_mw: 0 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 1 max_mcast_grp: 8192 max_mcast_qp_attach: 56 max_total_mcast_qp_attach: 458752 max_ah: 0 max_fmr: 0 max_srq: 65472 max_srq_wr: 16383 max_srq_sge: 31 max_pkeys: 128 local_ca_ack_delay: 15 port: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 4 port_lid: 8 port_lmc: 0x00 max_msg_sz: 0x40000000 port_cap_flags: 0x02510868 max_vl_num: 8 (4) bad_pkey_cntr: 0x0 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 128 gid_tbl_len: 128 subnet_timeout: 18 init_type_reply: 0 active_width: 4X (2) active_speed: 10.0 Gbps (4) phys_state: LINK_UP (5) GID[ 0]: fe80:0000:0000:0000:0025:90ff:ff07:0acd [cut] [cut] [fsimula@agape5 ~]$ sudo /usr/bin/ibv_devinfo -v hca_id: mlx4_0 transport: InfiniBand (0) fw_ver: 2.7.000 node_guid: 0025:90ff:ff07:0ac4 sys_image_guid: 0025:90ff:ff07:0ac7 vendor_id: 0x02c9 vendor_part_id: 26428 hw_ver: 0xB0 board_id: SM_1061000001000 phys_port_cnt: 1 max_mr_size: 0xffffffffffffffff page_size_cap: 0xfffffe00 max_qp: 261824 max_qp_wr: 16351 device_cap_flags: 0x006c9c76 max_sge: 32 max_sge_rd: 0 max_cq: 65408 max_cqe: 4194303 max_mr: 524272 max_pd: 32764 max_qp_rd_atom: 16 max_ee_rd_atom: 0 max_res_rd_atom: 4189184 max_qp_init_rd_atom: 128 max_ee_init_rd_atom: 0 atomic_cap: ATOMIC_HCA (1) max_ee: 0 max_rdd: 0 max_mw: 0 max_raw_ipv6_qp: 0 max_raw_ethy_qp: 1 max_mcast_grp: 8192 max_mcast_qp_attach: 56 max_total_mcast_qp_attach: 458752 max_ah: 0 max_fmr: 0 max_srq: 65472 max_srq_wr: 16383 max_srq_sge: 31 max_pkeys: 128 local_ca_ack_delay: 15 port: 1 state: PORT_ACTIVE (4) max_mtu: 2048 (4) active_mtu: 2048 (4) sm_lid: 4 port_lid: 6 port_lmc: 0x00 max_msg_sz: 0x40000000 port_cap_flags: 0x02510868 max_vl_num: 8 (4) bad_pkey_cntr: 0x0 qkey_viol_cntr: 0x0 sm_sl: 0 pkey_tbl_len: 128 gid_tbl_len: 128 subnet_timeout: 18 init_type_reply: 0 active_width: 4X (2) active_speed: 10.0 Gbps (4) phys_state: LINK_UP (5) GID[ 0]: fe80:0000:0000:0000:0025:90ff:ff07:0ac5 [cut] Il 2013-01-19 19:12 Mike Dubman ha scritto: > Hi Francesco, > Can you please provide complete output from ibv_devinfo -v command? > Also, it seems that you have Centos 5.8 with mxm/centos5.7 installed, will check if there is a distro version incompatibilities which may cause it and update you. > > Alina/Josh - please follow. > > Regards > M > > On Thu, Jan 17, 2013 at 4:09 PM, Francesco Simula <francesco.sim...@roma1.infn.it> wrote: > >> I tried building from OMPI 1.6.3 tarball with the following ./configure: >> ./configure --prefix=/apotto/home1/homedirs/fsimula/Lavoro/openmpi-1.6.3/install/ >> --disable-mpi-io >> --disable-io-romio >> --enable-dependency-tracking >> --without-slurm >> --with-platform=optimized >> --disable-mpi-f77 >> --disable-mpi-f90 >> --with-openib >> --disable-static >> --enable-shared >> --disable-vt >> --enable-pty-support >> --enable-mca-no-build=btl-ofud,pml-bfo >> --with-mxm=/opt/mellanox/mxm >> --with-mxm-libdir=/opt/mellanox/mxm/lib >> >> As you can see from the last two lines, I want to enable the MXM transport layer on a cluster made of SuperMicro X8DTG-D boards with dual Xeons and Mellanox MT26428 HCAs; the OS is CentOS 5.8. >> >> I tried with two different .rpm's for MXM, either 'mxm-1.1.ad085ef-1.x86_64-centos5u7.rpm' taken from here: >> http://www.mellanox.com/downloads/hpc/mxm/v1.1/mxm-latest.tar [1] >> >> and 'mxm-1.5.f583875-1.x86_64-centos5u7.rpm' taken from here: >> http://www.mellanox.com/downloads/hpc/mxm/v1.5/mxm-latest.tar [2] >> >> With both, even if the compilation concludes successfully, a simple test (osu_bw from the OSU Micro-Benchmarks 3.8) fails with the sort of message reported below; the lines: >> >> rdma_dev.c:122 MXM DEBUG Port 1 on mlx4_0 has a link layer different from IB. Skipping it >> rdma_dev.c:155 MXM ERROR An active IB port on a Mellanox device, with lid [any] gid [any] not found >> >> make it seem like it cannot access the HW for the HCA: is that so? The very same test works when using '-mca pml ob1' (thus using the openib BTL). >> >> I'm quite ready to start pulling my hair; any suggestions? >> >> The output of /usr/bin/ibv_devinfo for the two cluster nodes follows: >> [cut] >> hca_id: mlx4_0 >> transport: InfiniBand (0) >> fw_ver: 2.7.000 >> node_guid: 0025:90ff:ff07:0ac4 >> sys_image_guid: 0025:90ff:ff07:0ac7 >> vendor_id: 0x02c9 >> vendor_part_id: 26428 >> hw_ver: 0xB0 >> board_id: SM_1061000001000 >> phys_port_cnt: 1 >> port: 1 >> state: PORT_ACTIVE (4) >> max_mtu: 2048 (4) >> active_mtu: 2048 (4) >> sm_lid: 4 >> port_lid: 6 >> port_lmc: 0x00 >> [/cut] >> >> [cut] >> hca_id: mlx4_0 >> transport: InfiniBand (0) >> fw_ver: 2.7.000 >> node_guid: 0025:90ff:ff07:0acc >> sys_image_guid: 0025:90ff:ff07:0acf >> vendor_id: 0x02c9 >> vendor_part_id: 26428 >> hw_ver: 0xB0 >> board_id: SM_1061000001000 >> phys_port_cnt: 1 >> port: 1 >> state: PORT_ACTIVE (4) >> max_mtu: 2048 (4) >> active_mtu: 2048 (4) >> sm_lid: 4 >> port_lid: 8 >> port_lmc: 0x00 >> [/cut] >> >> The complete output of the failing test follows: >> >> [fsimula@agape5 osu-micro-benchmarks-3.8]$ mpirun -x MXM_LOG_LEVEL=poll -mca pml cm -mca mtl_mxm_np 1 -np 2 -host agape4,agape5 install/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_bw H H >> [1358430343.266782] [agape5:8596 :0] config_parser.c:168 MXM DEBUG >> [1358430343.266815] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_HANDLE_ERRORS=bt >> [1358430343.266826] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_GDB_PATH=/usr/bin/gdb >> [1358430343.266838] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_DUMP_SIGNO=1 >> [1358430343.266851] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_DUMP_LEVEL=conn >> [1358430343.266924] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_ASYNC_MODE=THREAD >> [1358430343.266936] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_TIME_ACCURACY=0.1 >> [1358430343.266956] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_PTLS=self,shm,rdma >> [1358430343.267249] [agape5:8596 :0] mpool.c:265 MXM DEBUG mpool 'ptl_self_recv_ev': allocated chunk 0xc075f40 of 96016 bytes with 1000 elements >> [1358430343.267308] [agape5:8596 :0] mpool.c:156 MXM DEBUG mpool 'ptl_self_recv_ev': align 16, maxelems 1000, elemsize 88, padding 8 >> [1358430343.267316] [agape5:8596 :0] self.c:410 MXM DEBUG Created ptl_self >> [1358430343.267333] [agape5:8596 :0] shm_ptl.c:56 MXM DEBUG Created ptl_shm >> [1358430343.268457] [agape5:8596 :0] rdma_ptl.c:65 MXM TRACE Got 1 IB devices >> [1358430343.268640] [agape5:8596 :0] rdma_ptl.c:112 MXM DEBUG added device mlx4_0 >> [1358430343.268665] [agape5:8596 :0] memreg.c:187 MXM TRACE Created memory registration cache on 1 devices >> [1358430343.268676] [agape5:8596 :0] rdma_ptl.c:133 MXM DEBUG Created ptl_rdma >> [1358430343.268689] [agape5:8596 :0] event.c:353 MXM FUNC mxm_event_init(event=0x2b73e0ee3038 mode=2 time_accuracy=160000000) >> [1358430343.268698] [agape5:8596 :0] timerq.c:55 MXM FUNC mxm_timerq_init(timerq=0x2b73e0ee3060 accuracy=160000000 max_interval=1600000000) >> [1358430343.268706] [agape5:8596 :0] event.c:292 MXM FUNC mxm_event_add_thread_context(thread=0x2b73e0ee30d0) >> [1358430343.268732] [agape5:8596 :0] event.c:198 MXM FUNC mxm_set_fd_nonblock(fd=10) >> [1358430343.268741] [agape5:8596 :0] event.c:198 MXM FUNC mxm_set_fd_nonblock(fd=11) >> [1358430343.268841] [agape5:8596 :0] mxm.c:162 MXM INFO context 0x2b73e0ee3010 created >> [1358430343.269090] [agape5:8596 :1] event.c:41 MXM FUNC __call_handler(handler->cb=0x2b73e0ab28a0 handler->arg=0x2b73e0ee3038) >> [1358430343.269104] [agape5:8596 :1] timerq.c:88 MXM FUNC mxm_timerq_sweep(timerq=0x2b73e0ee3060 current_time=568595527963578) >> [1358430343.274685] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_ENABLE_HUGETLB=1 >> [1358430343.274700] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_ENABLE_TIMEOUTS=y >> [1358430343.274709] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_ACK_TIMEOUT=0.3 >> [1358430343.274721] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_POLL_INTERVAL=0.1 >> [1358430343.274742] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_WINDOW_SIZE=512 >> [1358430343.274755] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_TX_BATCH=1 >> [1358430343.274764] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_CQ_MODERATION=64 >> [1358430343.274773] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_DRAIN_CQ=n >> [1358430343.274782] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_RNDV_THRESH=65536 >> [1358430343.274791] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_ZCOPY_THRESH=2040 >> [1358430343.274815] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_RESIZE_CQ=y >> [1358430343.274826] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_MTU=65536 >> [1358430343.274836] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_RX_QUEUE_LEN=16000 >> [1358430343.274849] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_TX_QUEUE_LEN=64 >> [1358430343.274859] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_RX_MAX_BUFFERS=128000 >> [1358430343.274877] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_TX_MAX_BUFFERS=8192 >> [1358430343.274887] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_RX_DROP_RATE=0 >> [1358430343.274896] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_ENABLE_NAK=y >> [1358430343.274904] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_RX_FILL_THRESH=0.6 >> [1358430343.274915] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_UD_TX_MAX_INLINE=128 >> [1358430343.274925] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_SHM_RX_MAX_BUFFERS=2000 >> [1358430343.274941] [agape5:8596 :0] config_parser.c:168 MXM DEBUG default: MXM_RDMA_ALLOC=1 >> [1358430343.274968] [agape5:8596 :0] ep.c:36 MXM FUNC mxm_ep_create(context=0x2b73e0ee3010) >> [1358430343.274984] [agape5:8596 :0] self.c:380 MXM DEBUG Created ptl_self EP(rank=3767085072) >> [1358430343.275028] [agape5:8596 :0] shm_queue.c:230 MXM DEBUG shm_ep=0, shmid=6815750 >> [1358430343.275072] [agape5:8596 :0] mpool.c:265 MXM DEBUG mpool 'shm_ep_recv': allocated chunk 0x2aaaadd0c010 of 65824016 bytes with 2000 elements >> [1358430343.278550] [agape5:8596 :0] mpool.c:156 MXM DEBUG mpool 'shm_ep_recv': align 16, maxelems 2000, elemsize 32904, padding 8 >> [1358430343.278584] [agape5:8596 :0] timerq.c:139 MXM FUNC mxm_timer_schedule(timerq=0x2b73e0ee3060 timer=0xc029538 expiration=568595550657300) >> [1358430343.278594] [agape5:8596 :0] timerq.c:43 MXM FUNC mxm_timerq_insert_timer(put timer 0xc029538 expiration 568595550657300 in slot 10) >> [1358430343.278608] [agape5:8596 :0] timerq.c:145 MXM TRACE added timer 0xc029538 expiration 568595550657300 interval 160000000 >> [1358430343.278617] [agape5:8596 :0] shm_ep.c:176 MXM DEBUG Created ptl_shm EP (rank=0, ctx_id=1) >> [1358430343.278641] [agape5:8596 :0] rdma_ep.c:317 MXM FUNC mxm_rdma_ep_create() >> [1358430343.278722] [agape5:8596 :0] rdma_dev.c:194 MXM FUNC mxm_rdma_dev_init(dev=0xc0b3f00) >> [1358430343.278924] [agape5:8596 :0] rdma_dev.c:122 MXM DEBUG Port 1 on mlx4_0 has a link layer different from IB. Skipping it >> [1358430343.278939] [agape5:8596 :0] rdma_dev.c:155 MXM ERROR An active IB port on a Mellanox device, with lid [any] gid [any] not found >> [1358430343.278954] [agape5:8596 :0] timerq.c:150 MXM FUNC mxm_timer_cancel(timerq=0x2b73e0ee3060 timer=0xc029538) >> [1358430343.279454] [agape5:8596 :0] mpool.c:184 MXM DEBUG mpool 'shm_ep_recv': destroyed >> [1358430343.279466] [agape5:8596 :0] self.c:287 MXM FUNC mxm_self_ep_destroy(ep=0xc094600) >> -------------------------------------------------------------------------- >> MXM was unable to create an endpoint. Please make sure that the network link is >> active on the node and the hardware is functioning. >> >> Error: No such device >> >> -------------------------------------------------------------------------- >> [1358430343.287336] [agape5:8596 :0] event.c:400 MXM FUNC mxm_event_cleanup(event=0x2b73e0ee3038) >> [1358430343.287348] [agape5:8596 :0] event.c:338 MXM FUNC mxm_event_remove_thread_context(thread=0x2b73e0ee30d0) >> [1358430343.287355] [agape5:8596 :0] event.c:145 MXM FUNC mxm_event_thread_wakeup() >> [1358430343.371011] [agape5:8596 :0] timerq.c:76 MXM FUNC mxm_timerq_cleanup(timerq=0x2b73e0ee3060) >> [1358430343.371030] [agape5:8596 :0] memreg.c:194 MXM TRACE Destroying memory registration cache >> [1358430343.371129] [agape5:8596 :0] shm_ptl.c:34 MXM FUNC ptl_shm_destroy(ptl=0xc0729b0) >> [1358430343.371139] [agape5:8596 :0] self.c:340 MXM FUNC mxm_self_destroy(ptl=0xc0699a0) >> [1358430343.371148] [agape5:8596 :0] mpool.c:184 MXM DEBUG mpool 'ptl_self_recv_ev': destroyed >> [1358430343.371156] [agape5:8596 :0] mxm.c:197 MXM INFO context 0x2b73e0ee3010 destroyed >> -------------------------------------------------------------------------- >> No available pml components were found! >> >> This means that there are no components of this type installed on your >> system or all the components reported that they could not be used. >> >> This is a fatal error; your MPI process is likely to abort. Check the >> output of the "ompi_info" command and ensure that components of this >> type are available on your system. You may also wish to check the >> value of the "component_path" MCA parameter and ensure that it has at >> least one directory that contains valid MCA components. >> -------------------------------------------------------------------------- >> [agape5:08596] PML cm cannot be selected >> -------------------------------------------------------------------------- >> mpirun has exited due to process rank 1 with PID 8596 on >> node agape5 exiting improperly. There are two reasons this could occur: >> >> 1. this process did not call "init" before exiting, but others in >> the job did. This can cause a job to hang indefinitely while it waits >> for all processes to call "init". By rule, if one process calls "init", >> then ALL processes must call "init" prior to termination. >> >> 2. this process called "init", but exited without calling "finalize". >> By rule, all processes that call "init" MUST call "finalize" prior to >> exiting or it will be considered an "abnormal termination" >> >> This may have caused other processes in the application to be >> terminated by signals sent by mpirun (as reported here). >> -------------------------------------------------------------------------- >> >> Regards, >> Francesco >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users [3] Links: ------ [1] http://www.mellanox.com/downloads/hpc/mxm/v1.1/mxm-latest.tar [2] http://www.mellanox.com/downloads/hpc/mxm/v1.5/mxm-latest.tar [3] http://www.open-mpi.org/mailman/listinfo.cgi/users
