For the sake of completeness, let me put the answer here. I posted the question on libfabric mailing list and with their input, installed librdmacm-devel. After that and a rebuild, the issue went away.
Thanks Durga We learn from history that we never learn from history. On Mon, Apr 4, 2016 at 3:20 PM, dpchoudh . <dpcho...@gmail.com> wrote: > Hi Howard > > Thank you very much for your suggestions. All the installation location in > my case are the default ones, so that is likely not the issue. > > What I find a bit confusing is this: > > As I mentioned, my cluster has both Qlogic Infiniband and Chelsio iWARP > (which are exposed to OpenMPI natively as well as an IP interface) > > With this configuration, if I build libfabric with configure options > --with-psm=auto --with-verbs=auto, as I mentioned earlier, only the PSM > interface shows up in the fi_info listing, and OpenMPI programs using ofi > MTL *do* work. However, I do not know if the traffic is going through the > Qlogic card or the Chelsio card; it is likely the former. > > I am going to ask this on the libfabric list, but perhaps the following > question is relevant in the OpenMPI list: > > My understanding about the OFI MTL is the following: please correct me > where I am wrong: > As I understand, ANY type of transport that exposes a verbs interface > (iWARP, RoCE, Infiniband from any manufacturer) can become a libfabric > provider (when libfabric is compiled with --with-verbs option) and thus > support the OFI MTL (and thus the cm PML?) > > Is the above true? > > Best regards > Durga > > We learn from history that we never learn from history. > > On Mon, Apr 4, 2016 at 7:29 AM, Howard Pritchard <hpprit...@gmail.com> > wrote: > >> Hi Durga, >> >> I'd suggest reposting this to the libfabric-users mail list. >> You can join that list at >> http://lists.openfabrics.org/mailman/listinfo/libfabric-users >> >> I'd suggest including the output of config.log. If you installed >> ofed in non-canonical location, you may need to give an explicit >> path as an argument to the --enable-verbs configury option. >> >> Note if you're trying to use libfabric with the Open MPI ofi >> mtl, you will need to get literally the freshest version of >> libfabric, either at github or the 1.3rc2 tarball at >> >> http://www.openfabrics.org/downloads/ofi/ >> >> Good luck, >> >> Howard >> >> >> 2016-04-02 13:41 GMT-06:00 dpchoudh . <dpcho...@gmail.com>: >> >>> Hello all >>> >>> My machine has 3 network cards: >>> >>> 1. Broadcom GbE (vanilla type, with some offload capability) >>> 2. Chelsion S310 10Gb iWARP >>> 3. Qlogic DDR 4X Infiniband. >>> >>> With this setup, I built libfabric like this: >>> >>> ./configure --enable-udp=auto --enable-gni=auto --enable-mxm=auto >>> --enable-usnic=auto --enable-verbs=auto --enable-sockets=auto >>> --enable-psm2=auto --enable-psm=auto && make && sudo make install >>> >>> However, in the built libfabric, I do not see a verb provider, which I'd >>> expect for the iWARP card, at least. >>> >>> [durga@smallMPI libfabric]$ fi_info >>> psm: psm >>> version: 0.9 >>> type: FI_EP_RDM >>> protocol: FI_PROTO_PSMX >>> UDP: UDP-IP >>> version: 1.0 >>> type: FI_EP_DGRAM >>> protocol: FI_PROTO_UDP >>> sockets: IP >>> version: 1.0 >>> type: FI_EP_MSG >>> protocol: FI_PROTO_SOCK_TCP >>> sockets: IP >>> version: 1.0 >>> type: FI_EP_DGRAM >>> protocol: FI_PROTO_SOCK_TCP >>> sockets: IP >>> version: 1.0 >>> type: FI_EP_RDM >>> protocol: FI_PROTO_SOCK_TCP >>> >>> >>> Am I doing something wrong or misunderstanding how libfabric works? >>> >>> Thanks in advance >>> Durga >>> >>> We learn from history that we never learn from history. >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/04/28870.php >>> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/04/28884.php >> > >
