For the GPU Direct to work with Infiniband, you need to get some updated OFED bits from your Infiniband vendor.
In terms of checking the driver updates, you can do a grep on the string get_driver_pages in the file/proc/kallsyms. If it is there, then the Linux kernel is updated correctly. The GPU Direct functioning should be independent of the MPI you are using. Rolf -----Original Message----- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Brice Goglin Sent: Monday, February 28, 2011 11:42 AM To: us...@open-mpi.org Subject: Re: [OMPI users] anybody tried OMPI with gpudirect? Le 28/02/2011 17:30, Rolf vandeVaart a écrit : > Hi Brice: > Yes, I have tired OMPI 1.5 with gpudirect and it worked for me. You > definitely need the patch or you will see the behavior just as you described, > a hang. One thing you could try is disabling the large message RDMA in OMPI > and see if that works. That can be done by adjusting the openib BTL flags. > > -- mca btl_openib_flags 304 > > Rolf > Thanks Rolf. Adding this mca parameter worked-around the hang indeed. The kernel is supposed to be properly patched for gpudirect. Are you aware of anything else we might need to make this work? Do we need to rebuild some OFED kernel modules for instance? Also, is there any reliable/easy way to check if gpudirect works in our kernel ? (we had to manually fix the gpudirect patch for SLES11). Brice _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
