On 24 October 2007 at 21:31, Jeff Squyres wrote:
| On Oct 24, 2007, at 9:23 PM, Dirk Eddelbuettel wrote:
|
| > | If I had to guess, the systems where you don't see the warning are
| > | systems that have OFED loaded.
| >
| > I am pretty sure that none of the systems (at work) have IB
| > hardware. I am
| > very sure that my home systems do not, and there the 'btl = ^openib'
| > successfully suppresses the warning --- whereas at work it doesn't.
|
| Note that you don't need to have IB hardware -- all you need is the
| OFED software loaded. I don't know if Debian ships the OFED
| libraries by default...? In particular, look for libibverbs:
|
| [18:28] svbu-mpi:~/svn/ompi % ldd $bogus/lib/openmpi/mca_btl_openib.so
| libibverbs.so.1 => /usr/lib64/libibverbs.so.1
| (0x0000002a956c2000)
| libnsl.so.1 => /lib64/libnsl.so.1 (0x0000002a957cd000)
| libutil.so.1 => /lib64/libutil.so.1 (0x0000002a958e4000)
| libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a959e8000)
| libpthread.so.0 => /lib64/tls/libpthread.so.0
| (0x0000002a95b6e000)
| libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a95c83000)
| libdl.so.2 => /lib64/libdl.so.2 (0x0000002a95eb8000)
| /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
Good point. However, I use the .deb packages which are I build for Debian,
and they use libibverbs where available:
Build-Depends: [...], libibverbs-dev [!kfreebsd-i386 !kfreebsd-amd64 \
!hurd-i386], gfortran, libsysfs-dev, automake, gcc (>= 4:4.1.2)
in particular on i386. Consequently, the binary package ends up with a
Depends on the run-time package 'libibverbs1' -- and this will hence always
be present as all my systems use the .deb packages (either from Debian or
locally rebuild) that forces libibverbs1 in via this Depends.
At work, I re-build these same package under Ubuntu on my "head node". And
on the head node, no warning is seen -- wherease my computes issue the
warning.
Could this be another one of the dlopen issues where basically
ldopen("libibverbs.so")
is executed? Because the compute nodes do NOT have libibverbs.so (from the
-dev package) but only libibverbs.so.1.0.0 and its matching symlink
libibverbs.so.1.
I just tested that hypothesis and install libibverbs-dev, but no beans. Still
get the warning.
| However, I note something in your last reply that I may have missed
| before -- can you clarify a point for me: are you saying that on your
| home machine, this generates the openib "file not found" warning:
|
| mpirun -np 2 hello
|
| but this does not:
|
| mpirun -np 2 --mca btl ^openib hello
More or less, but I use /etc/openmpi/openmci-mca-params.conf to toggle
^openib. Adding it again as --mca btl ^openib changes nothing, unfortunately.
| If so, can you confirm which version of Open MPI you are running?
| The only reason that I can think that that would happen is if you are
| running a trunk nightly download of Open MPI... If not, then there's
| something else going on that would be worth understanding.
No, plain 1.2.4 from the original tarballs.
Still puzzled. To recap, the head node and the compute node all use the same
Ubuntu release, use the same binary .deb packages from Open MPI 1.2.4 I
rebuild there. The 'sole' difference is that the 'head node' has more
development packages and tools installed -- but that should not matter. I
just re-checked and the compute node does not have any LAM or MPICH
parts remaining.
Dirk
--
Three out of two people have difficulties with fractions.
