Hi all, I have been trying to setup a cluster with QLogic QLE7140 HCA's and a Cisco SFS-7000 24 port switch. The machines are running Debian Wheezy.
I have installed OpenMPI from the repos (1.4.5) and also libibverbs1 libipathverbs1 libmthca1 librdmacm1 I have also tested OpenMPI compiled from the latest sources (1.6.4) with the same results. I modprobe rdma_ucm, ib_umad and ib_uverbs in order to get MPI jobs to run. I'm not actually sure if what I've done is enough to correctly configure the network but I have tested several MPI capable codes that use Fortran and C, specifying the openib interface with the flag '--mca btl openib,self'. Things initially work and the bandwidth is as expected, however after anywhere from 4 to 30 hours the jobs crash. The longest job that has completed successfully has gone for around 48 hours however they rarely make it past 4 hours. This is always the error message. [[36446,1],2][../../../../../../ompi/mca/btl/openib/btl_openib_component.c:3238:handle_wc] from host84 to: host85 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 36085024 opcode 2 vendor error 0 qp_idx 3 -------------------------------------------------------------------------- The OpenFabrics stack has reported a network error event. Open MPI will try to continue, but your job may end up failing. Local host: host85 MPI process PID: 7912 Error number: 10 (IBV_EVENT_PORT_ERR) This error may indicate connectivity problems within the fabric; please contact your system administrator. -------------------------------------------------------------------------- The InfiniBand retry count between two MPI processes has been exceeded. "Retry count" is defined in the InfiniBand spec 1.2 (section 12.7.38): The total number of times that the sender wishes the receiver to retry timeout, packet sequence, etc. errors before posting a completion error. This error typically means that there is something awry within the InfiniBand fabric itself. You should note the hosts on which this error has occurred; it has been observed that rebooting or removing a particular host from the job can sometimes resolve this issue. Two MCA parameters can be used to control Open MPI's behavior with respect to the retry count: * btl_openib_ib_retry_count - The number of times the sender will attempt to retry (defaulted to 7, the maximum value). * btl_openib_ib_timeout - The local ACK timeout parameter (defaulted to 20). The actual timeout value used is calculated as: 4.096 microseconds * (2^btl_openib_ib_timeout) See the InfiniBand spec 1.2 (section 12.7.34) for more details. Below is some information about the host that raised the error and the peer to which it was connected: Local host: host85 Local device: qib0 Peer host: host84 You may need to consult with your system administrator to get this problem fixed. -------------------------------------------------------------------------- On a few occasions the machine that initiated the failure, (host85 in this example) crashed to the point of needing to be power cycled however most times, only Infiniband connectivity was lost after the crash. I have checked kernel and system logs and can't find anything at the time of the crash. I have seen it recommended to use psm instead of openib for QLogic cards. Could this be part of the problem? After a lot of experimentation I am at a complete loss as to how to get psm up and running. If possible, could someone also help me understand which out of this list (ibibverbs, libipathverbs, libmthca, librdmacm, ib_mad, ib_umad, rdma_ucm, ib_uverbs, ib_qib) is the actual driver for my card and is there any way to configure the driver? This blog posthttp://swik.net/Debian/Planet+Debian/Julien+Blache%3A+QLogic+QLE73xx+InfiniBand+adapters,+QDR,+ib_qib,+OFED+1.5.2+and+Debian+Squeeze/e56if seems to suggest that I will need to download the complete QLogic OFED stack and configure the driver which I've tried to do and failed on many counts. I would be very grateful for any advice at this stage. Best regards, Vanja
