I would recommend you to read OFED (or Mellanox OFED) documentation. It will be good start point.
Regards, Pavel (Pasha) Shamis --- Application Performance Tools Group Computer Science and Math Division Oak Ridge National Laboratory On Mar 14, 2011, at 4:37 PM, Bernardo F Costa wrote: > Ok. Native ibverbs/openib is preferable although cannot be used by all > applications (those who do not have a native ip interface). I suppose > that if I configure my network nodes to use ipoib (by simply probing > ib_ipoib module) I'd still be able to use native ibverbs interface > without any delay caused by ipoib on it. And by this way other > applications which aren't able to use native ibverbs could use the > infiniband network as well. It should be the reason why some people > use ipoib I believe, just to offer infiniband to all network > applications. The main reason I've asked this question is that I have > seen lots of references in the net on how to configure an infiniband > network with ipoib, but I was not able to see much references on doing > the same without ipoib. This made me believe configuring infiniband > with ipoib could be a popular option. > I've tried ibdiagnet and other ofed tools. I also tried to debug the > network environment with simple jobs to measure bandwidth and latency. > In most cases, I've seen high peaks of measures who come and go > without any reason I could catch for now. I believe I should check the > network configuration and make some tests on it. Does anybody here > know some reference about configuring inifiband without ipoib and/or > the issues raised when doing this ? If possible, I'd like to see ways > of testing the configuration, or know about options that could > increase fault tolerance. I know this is some kind of basic, but I am > not a well experienced user on infiniband. > > 2011/3/14 <users-requ...@open-mpi.org>: >> >> Please see my comment below. >> >> Pavel (Pasha) Shamis >> --- >> Application Performance Tools Group >> Computer Science and Math Division >> Oak Ridge National Laboratory >> >> >> On Mar 11, 2011, at 2:47 PM, Bernardo F Costa wrote: >> >>> I have found this thread form two years ago. I am some kind of lost on >>> configuring an infiniband cluster for openmpi. What is best: use iboip >>> or use native infiniband ibverbs interface ? For now I am using native >> >> Native openib/verbs interface will work much faster (up X10) then ipoib. >> Ipoib was designed for application that does not have native ip interface/ >> >>> infiniband withou ipoib. But I have lots of problems specially with >>> latency in the cluster. >> >> If you see latency problems over native interface (verbs), then you >> apparently will >> face the same problem with any other application over verbs, including ipoib. >> >> So using ipoib instead of verbs definitely not a work around for you. >> >> I would suggest you to run IB network debug tools , like ibdiagnet , in >> order to analyze >> your network/latency problems. >> >> Regards, >> Pasha > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
