Simon Su <newsgroup4...@gmail.com> writes: > Hi Tom, > > "hello world" MPI program also won't compile if > librdmacm-devel-1.0.8-5.el5 is not installed. I have asked the person > who maintain the openmpi package on how they were compiled. My guess > is librdmacm-devel-1.0.8-5.el5 may need to be added as dependency > package for openmpi010208-gcc-devel-1.2.8-8.cses.5.PU_IAS.5 package > (where I got my openmpi installation) to solve the problem and to > verify that we have the correct openmpi compilation.
Yes, I agree, from this message && your last (to OMPI: Simon mentioned that the issue goes away if he installs librdmacm-devel), it sounds like librdmacm-devel is a "build-dep", but the openmpi-devel package needs it as a "dep". Anyway, I'll leave it up to you to forward the error/conclusion to whomever your upstream is. Thanks for digging into this, -tom > On Wed, Dec 16, 2009 at 5:45 PM, tom fogal <tfo...@sci.utah.edu> wrote: > > > Hi, > > > > Jeff sent this reply to our inquiry yesterday. > > > > Simon -- can you give it a read? In particular, validating you've got > > the right mpic++ sounds like a good idea. We're also curious if a > > simple "hello world" MPI program can link using gcc + the flags from > > mpic++ -show. > > > > -tom > > > > ------- Forwarded Message > > > > From: Jeff Squyres <jsquy...@cisco.com> > > In-Reply-To: <auto-000021432...@sci.utah.edu> > > Date: Wed, 16 Dec 2009 17:20:35 -0500 > > To: "Open MPI Users" <us...@open-mpi.org> > > Cc: VisIt Developers <visit-develop...@email.ornl.gov> > > Subject: Re: [OMPI users] [visit-developers] /usr/bin/ld: cannot > > find-lrdmacm on 9184 > > > > It depends on how Open MPI was built. > > > > If Open MPI was built without plugins (i.e., all the plugins are slurped up > > into libmpi and friends), then yes, applications need to link against > > librdmacm to use the RDMA CM mode of OpenFabrics transport. > > > > If Open MPI was built with plugins (which is the default), then apps don't > > need to link against librdmacm because the only use of rdmacm is in an Open > > MPI plugin, and that plugin was linked against librdmacm. > > > > Make sense? > > > > That being said, the output from mpic++ --showme should give you something > > that is directly compile-/link-able. So it is odd if mpic++ is showing you > > something that can't (or shouldn't?) be done. Did a -L argument get lost > > somewhere, perchance? > > > > Does linking MPI applications with mpic++ work properly, or does it result > > in the same error? If it results in the same error, then perhaps something > > has changed since Open MPI was installed...? > > > > All this being said, two other random points: > > > > 1. Ensure that you're using the "right" mpic++. I.e., make sure it matches > > the version/installation of Open MPI that you're trying to use. > > > > 2. If you don't link with the librdmacm, you're probably not losing any > > important functionality unless you have an iWarp-based cluster (that's the > > only transport that *needs* librdmacm). IB-based networks can use > > librdmacm, but don't *need* it (it's only used for making initial > > connections, so using librdmacm or not has no implications on overall MPI > > performance). It's still odd that mpic++ wants it and it can't be found, > > though... > > > > Does that helps? > > > > > > On Dec 15, 2009, at 11:11 PM, tom fogal wrote: > > > > > Simon Su <newsgroup4...@gmail.com> writes: > > > > Hi Tom, > > > > > > > > I am using the standard openmpi package that run on all the cluster > > > > machines here at Princeton. So, maybe I shouldn't touch openmpi. But, > > > > removing -lrdmacm from the MPI_LIBS line in the machinename.conf file > > > > worked. Any implication from doing this? > > > > > > The only thing it could possibly do is disable RDMA for you. However, > > > since removing it did not produce any undefined symbol errors, my guess > > > is that your OpenMPI isn't using RDMA anyway. > > > > > > There might be an OpenMPI bug here, though. I've cc'd the OpenMPI > > > community to see if they have any input. As a summary for them: Simon > > > is trying to build our MPI-enabled application. A script which tries > > > to automate this adds the output of "mpic++ -show". His build then > > > failed because it attempted to link against librdmacm, which does not > > > exist in his normal search paths (or maybe at all). Is it possible > > > that `mpic++ -show' includes/adds "-lrdmacm" even when OpenMPI is not > > > itself using the library? > > > > > > Thanks, > > > > > > -tom > > > > > > > On Tue, Dec 15, 2009 at 8:46 PM, tom fogal <tfo...@alumni.unh.edu> > > wrote: > > > > > > > > > Simon Su <newsgroup4...@gmail.com> writes: > > > > > > I am getting this error message while building 9184. > > > > > [snip] > > > > > > -lz -lm -ldl -lpthread -L/usr/local/openmpi/1.3.3/gcc/x86_64/lib64 > > > > > > -lmpi_cxx -lmpi -lopen-rte -lopen-pal -lrdmacm -libverbs -lnuma > > -ldl > > > > > -lnsl > > > > > > -lutil -lm -lcognomen \ > > > > > > -L/usr/local/openmpi/1.3.3/gcc/x86_64/lib64 -lmpi_cxx -lmpi > > > > > > -lopen-rte -lopen-pal -lrdmacm -libverbs -lnuma -ldl -lnsl -lutil > > -lm > > > > > > -lcognomen > > > > > > /usr/bin/ld: cannot find -lrdmacm > > > > > > collect2: ld returned 1 exit status > > > > > > > > > > Your OpenMPI install (incorrectly?) thinks it has librdmacm > > available, > > > > > but the library isn't in your search path. > > > > > > > > > > It apparently defaults to enabled in 1.3.3. That seems rather > > > > > silly, since I imagine the library requires RDMA hardware, which > > > > > is of course not ubiquitous. Anyway, try configuring OpenMPI with > > > > > --disable-openib-rdmacm and then rerunning build_visit. > > > > > > > > > > Of course, if you actually have an RDMA cluster, you'll want to delve > > > > > deeper. > > > > > > > > > > Cheers, > > > > > > > > > > -tom > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > > > > - -- > > Jeff Squyres > > jsquy...@cisco.com > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ------- End of Forwarded Message > > > > > > --001636e0b6b9dccd4d047af22bb9 > Content-Type: text/html; charset=ISO-8859-1 > Content-Transfer-Encoding: quoted-printable > > Hi Tom,<br><br>"hello world" MPI program also won't compile i= > f librdmacm-devel-1.0.8-5.el5 is not installed. I have asked the person who= > maintain the openmpi package on how they were compiled. My guess is librdm= > acm-devel-1.0.8-5.el5 may need to be added as dependency package for openmp= > i010208-gcc-devel-1.2.8-8.cses.5.PU_IAS.5 package (where I got my openmpi i= > nstallation) to solve the problem and to verify that we have the correct op= > enmpi compilation.<br> > <br>Thanks for your help<br>Simon<br><br><div class=3D"gmail_quote">On Wed,= > Dec 16, 2009 at 5:45 PM, tom fogal <span dir=3D"ltr"><<a href=3D"mailto= > :tfo...@sci.utah.edu">tfo...@sci.utah.edu</a>></span> wrote:<br><blockqu= > ote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204= > ); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> > Hi,<br> > <br> > Jeff sent this reply to our inquiry yesterday.<br> > <br> > Simon -- can you give it a read? =A0In particular, validating you've go= > t<br> > the right mpic++ sounds like a good idea. =A0We're also curious if a<br= > > > simple "hello world" MPI program can link using gcc + the flags f= > rom<br> > mpic++ -show.<br> > <br> > -tom<br> > <br> > ------- Forwarded Message<br> > <br> > From: Jeff Squyres <<a href=3D"mailto:jsquy...@cisco.com";>jsquyres@cisco= > .com</a>><br> > In-Reply-To: <<a href=3D"mailto:auto-000021432...@sci.utah.edu";>auto-000= > 021432...@sci.utah.edu</a>><br> > Date: Wed, 16 Dec 2009 17:20:35 -0500<br> > To: "Open MPI Users" <<a href=3D"mailto:us...@open-mpi.org";>us= > e...@open-mpi.org</a>><br> > Cc: VisIt Developers <<a href=3D"mailto:visit-develop...@email.ornl.gov"= > >visit-develop...@email.ornl.gov</a>><br> > Subject: Re: [OMPI users] [visit-developers] /usr/bin/ld: cannot<br> > =A0 =A0 =A0 =A0find-lrdmacm on 9184<br> > <br> > It depends on how Open MPI was built.<br> > <br> > If Open MPI was built without plugins (i.e., all the plugins are slurped up= > into libmpi and friends), then yes, applications need to link against libr= > dmacm to use the RDMA CM mode of OpenFabrics transport.<br> > <br> > If Open MPI was built with plugins (which is the default), then apps don= > 9;t need to link against librdmacm because the only use of rdmacm is in an = > Open MPI plugin, and that plugin was linked against librdmacm.<br> > <br> > Make sense?<br> > <br> > That being said, the output from mpic++ --showme should give you something = > that is directly compile-/link-able. =A0So it is odd if mpic++ is showing y= > ou something that can't (or shouldn't?) be done. =A0Did a -L argume= > nt get lost somewhere, perchance?<br> > > <br> > Does linking MPI applications with mpic++ work properly, or does it result = > in the same error? =A0If it results in the same error, then perhaps somethi= > ng has changed since Open MPI was installed...?<br> > <br> > All this being said, two other random points:<br> > <br> > 1. Ensure that you're using the "right" mpic++. =A0I.e., make= > sure it matches the version/installation of Open MPI that you're tryin= > g to use.<br> > <br> > 2. If you don't link with the librdmacm, you're probably not losing= > any important functionality unless you have an iWarp-based cluster (that&#= > 39;s the only transport that *needs* librdmacm). =A0IB-based networks can u= > se librdmacm, but don't *need* it (it's only used for making initia= > l connections, so using librdmacm or not has no implications on overall MPI= > performance). =A0It's still odd that mpic++ wants it and it can't = > be found, though...<br> > > <br> > Does that helps?<br> > <div><div></div><div class=3D"h5"><br> > <br> > On Dec 15, 2009, at 11:11 PM, tom fogal wrote:<br> > <br> > > Simon Su <<a href=3D"mailto:newsgroup4...@gmail.com";>newsgroup4ssu@= > gmail.com</a>> writes:<br> > > > Hi Tom,<br> > > ><br> > > > I am using the standard openmpi package that run on all the clust= > er<br> > > > machines here at Princeton. So, maybe I shouldn't touch openm= > pi. But,<br> > > > removing -lrdmacm from the MPI_LIBS line in the machinename.conf = > file<br> > > > worked. =A0Any implication from doing this?<br> > ><br> > > The only thing it could possibly do is disable RDMA for you. =A0Howeve= > r,<br> > > since removing it did not produce any undefined symbol errors, my gues= > s<br> > > is that your OpenMPI isn't using RDMA anyway.<br> > ><br> > > There might be an OpenMPI bug here, though. =A0I've cc'd the O= > penMPI<br> > > community to see if they have any input. =A0As a summary for them: Sim= > on<br> > > is trying to build our MPI-enabled application. =A0A script which trie= > s<br> > > to automate this adds the output of "mpic++ -show". =A0His b= > uild then<br> > > failed because it attempted to link against librdmacm, which does not<= > br> > > exist in his normal search paths (or maybe at all). =A0Is it possible<= > br> > > that `mpic++ -show' includes/adds "-lrdmacm" even when O= > penMPI is not<br> > > itself using the library?<br> > ><br> > > Thanks,<br> > ><br> > > -tom<br> > ><br> > > > On Tue, Dec 15, 2009 at 8:46 PM, tom fogal <<a href=3D"mailto:= > tfo...@alumni.unh.edu">tfo...@alumni.unh.edu</a>> wrote:<br> > > ><br> > > > > Simon Su <<a href=3D"mailto:newsgroup4...@gmail.com";>news= > group4...@gmail.com</a>> writes:<br> > > > > > I am getting this error message while building 9184.<br= > > > > > > [snip]<br> > > > > > -lz -lm -ldl =A0-lpthread -L/usr/local/openmpi/1.3.3/gc= > c/x86_64/lib64<br> > > > > > -lmpi_cxx -lmpi -lopen-rte -lopen-pal -lrdmacm -libverb= > s -lnuma -ldl<br> > > > > -lnsl<br> > > > > > -lutil -lm =A0-lcognomen \<br> > > > > > =A0 =A0 =A0 =A0 -L/usr/local/openmpi/1.3.3/gcc/x86_64/l= > ib64 -lmpi_cxx -lmpi<br> > > > > > -lopen-rte -lopen-pal -lrdmacm -libverbs -lnuma -ldl -l= > nsl -lutil -lm<br> > > > > > -lcognomen<br> > > > > > /usr/bin/ld: cannot find -lrdmacm<br> > > > > > collect2: ld returned 1 exit status<br> > > > ><br> > > > > Your OpenMPI install (incorrectly?) thinks it has librdmacm = > available,<br> > > > > but the library isn't in your search path.<br> > > > ><br> > > > > It apparently defaults to enabled in 1.3.3. =A0That seems ra= > ther<br> > > > > silly, since I imagine the library requires RDMA hardware, w= > hich<br> > > > > is of course not ubiquitous. =A0Anyway, try configuring Open= > MPI with<br> > > > > --disable-openib-rdmacm and then rerunning build_visit.<br> > > > ><br> > > > > Of course, if you actually have an RDMA cluster, you'll = > want to delve<br> > > > > deeper.<br> > > > ><br> > > > > Cheers,<br> > > > ><br> > > > > -tom<br> > </div></div>> _______________________________________________<br> > > users mailing list<br> > > <a href=3D"mailto:us...@open-mpi.org";>us...@open-mpi.org</a><br> > > <a href=3D"http://www.open-mpi.org/mailman/listinfo.cgi/users"; target= > =3D"_blank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br> > ><br> > <br> > <br> > - --<br> > Jeff Squyres<br> > <a href=3D"mailto:jsquy...@cisco.com";>jsquy...@cisco.com</a><br> > <br> > <br> > _______________________________________________<br> > users mailing list<br> > <a href=3D"mailto:us...@open-mpi.org";>us...@open-mpi.org</a><br> > <a href=3D"http://www.open-mpi.org/mailman/listinfo.cgi/users"; target=3D"_b= > lank">http://www.open-mpi.org/mailman/listinfo.cgi/users</a><br> > <br> > ------- End of Forwarded Message<br> > <br> > </blockquote></div><br> > > --001636e0b6b9dccd4d047af22bb9--
