[OMPI users] using ompi-server on a single node

[Thomas Ropars]

Hi,

I've tried to use ompi-server to connect 2 processes belonging to
different jobs but running on the same computer. It works when the
computer has a network interface up. But if the only active network
interface is the local loop, it doesn't work.

According to what I understood reading the code, it is because no btl
component can be used in this case. "tcp" is not used because usually
it is the "sm" component that is used for processes on the same host.
But in that case it doesn't work because "sm" is supposed to work only
for processes of the same job.

I know that this use-case is not very frequent  :) 


But Is there a solution to make it work ? or is it a known limitation ?

Regards

Thomas

Re: [OMPI users] using ompi-server on a single node

[Ralph Castain]

It is currently a known limitation - shared memory currently only  
works between procs from the same job. There is an enhancement coming  
that will remove this restriction, but it won't be out for some time.


Ralph

On Jan 5, 2009, at 1:06 AM, Thomas Ropars wrote:


Hi,

I've tried to use ompi-server to connect 2 processes belonging to
different jobs but running on the same computer. It works when the
computer has a network interface up. But if the only active network
interface is the local loop, it doesn't work.

According to what I understood reading the code, it is because no btl
component can be used in this case. "tcp" is not used because usually
it is the "sm" component that is used for processes on the same host.
But in that case it doesn't work because "sm" is supposed to work only
for processes of the same job.

I know that this use-case is not very frequent  :)
But Is there a solution to make it work ? or is it a known  
limitation ?


Regards

Thomas

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] using the carto facility

[Lydia Heck]

I was advised for a benchmark to use the OPAL carto option to
assign specific cores to a job. I searched the web for an example
but have only found one set of man pages, which is rather cryptic
and needs the knowledge of the programmer rather than an end user.

Has anybody out there used this option and if so would you be prepared
to share an example which could be adapted for a shared memory system
with silions of cores.

Thanks.

Lydia


--
Dr E L  Heck

University of Durham
Institute for Computational Cosmology
Ogden Centre
Department of Physics
South Road

DURHAM, DH1 3LE
United Kingdom

e-mail: lydia.h...@durham.ac.uk

Tel.: + 44 191 - 334 3628
Fax.: + 44 191 - 334 3645
___

Re: [OMPI users] question running on heterogeneous systems

[Gus Correa]

Mahmoud Payami wrote:



On Fri, Jan 2, 2009 at 9:08 AM, doriankrause > wrote:


Mahmoud Payami wrote:


Dear OpenMPI Users,

I have two systems, one with Intel64 processor, and one with
IA32. The OSs on first is CentOS-86_64 and the other
CentOS-i386. I installed Intel fortran compiler 10.1 on both.
 In the first I use the fce, and in the second I use fc
directories (ifortvars.sh/csh). I have compiled openmpi
separately on each machine. Now, I could not run my
application whch is compiled on ia32 machine. Should I use
"fc" instead of "fce" on intel64 and then compile openmpi with
that?


Could you give us some more information? What is the error message?
You said that the application is compiled for the 32 bit
architecture. I'm not used to mixing 32/64 bit architectures. Does
the application run on each host seperately?

Dorian




Hi Mahmoud, list

Dear Dorian,
Thank you  for your contribution. The application, compiled on each 
box separately, is ok with mpi an no problem. Recently, I had checked 
that a binary file created on ia32, also works on 86_64 but the 
reverse is not true.

That is correct.
x86-64 architecture can run 32-bit binaries,
but 64-bit binaries don't work on x86 machines.
So, why not a parallel program which is compiled on ia32 box? I think, 
if I configure and install openmpi using ia32 intel compiler on 86_64 
box, then it will be resolved.

1. You need to compile OpenMPI separately on each architecture.
Use the "--prefix=/path/to/my/openmpi/32bit/" (32-bit example/suggestion)
configure option, to install the two libraries on different locations,
if you want. 
This will make clear for which architecture the library was built for.


2. You need to compile your application separately on each architecture,
and link to the OpenMPI libraries built for that specific architecture
according to item 1  above.
(I.e. don't mix apples and oranges.)

3. You need to have the correct environment variables set
on each machine architecture.
They are *different* on each architecture.

I.e., if you use Intel Fortran,
source the fc script on the 32bit machine,
and source the fce script on the 64-bit machine.

This can be done on the .bashrc or .tcshrc file.
If you have a different home directory on each machine,
you can write a .bashrc or .tcshrc file for each architecture.
If you have a single NFS mounted home directory,
use a trick like this (tcsh example):

if ( $HOST == "my_32bit_hostname" ) then
   source /path/to/intel/fc/bin/ifortvars.csh # Note "fc" here.
else if ( $HOST == "my_64bit_hostname"  ) then
   source /path/to/intel/fce/bin/ifortvars.csh   # Note "fce" here.
endif

whatever your "my_32bit_hostname", "my_64bit_hostname".
/path/to/intel/fc/, and  /path/to/intel/fce/   are.
(Do "hostname" on each machine to find out the right name to use.)

Likewise for the OpenMPI binaries (mpicc, mpif90, mpirun, etc):

if ( $HOST == "my_32bit_hostname" ) then
   setenv PATH /path/to/my/openmpi/32bit/bin:$PATH   # Note "32bit" here.
else if ( $HOST == "my_64bit_hostname"  ) then
   setenv PATH /path/to/my/openmpi/64bit/bin:$PATH# Note "64bit" here.
endif

This approach also works for separate home directories "per machine"
(not NFS mounted), and is probably the simplest way to solve the problem.

There are more elegant ways to setup the environment of choice,
other than changing the user startup files.
For instance, you can write intel.csh and intel.sh on the /etc/profile.d 
directory,

to setup the appropriate environment as the user logs in.
See also the "environment modules" package:
http://modules.sourceforge.net/

4) If you run MPI programs across the two machines/architectures,
make sure to use the MPI types on MPI function calls correctly,
and to match them properly to the native Fortran (or C) types
on each machine/architecture.

I hope this helps.
Gus Correa
-
Gustavo Correa, PhD - Email: g...@ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
-
I have to check it and will report the result. In present case, it is 
searching for shared lib.so.0 which has some extension "..ELF...64". I 
have already added "/usr/local/lib" which contains mpi libs in 
LD_LIBRARY_PATH otherwise they would not work on each box even separatey.

Bests, Happy 2009
mahmoud
 



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] openMPI, transfer data from multiple sources to one destination

[Eugene Loh]

Jack Bryan wrote:

  I need to transfer data from multiple sources to one
destination. 
The requirement is:
  
(1) The sources and destination nodes may work asynchronously.
 
(2) Each source node generates data package in their own paces.
    And, there may be many packages to send. Whenever, a data package 
    is generated , it should be sent to the desination node at once.
    And then, the source node continue to work on generating the next 
    package. 
  
(3) There is only one destination node , which must receive all data 
    package generated from the source nodes. 
    Because the source and destination nodes may work asynchronously,
    the destination node should not wait for a specific source node
until 
    the source node sends out its data. 
  
    The destination node should be able to receive data package 
    from anyone source node whenever the data package is available in a
  
    source node. 
  
My question is :
  
What MPI function should be used to implement the protocol above ? 
  
If I use MPI_Send/Recv, they are blocking function. The destination
node have to wait for one node until its data is available. 
  
The communication overhead is too high. 
  
If I use MPI_Bsend, the destination node has to use MPI_Recv to , 
a Blocking receive for a message .
  
This can make the destination node wait for only one source node and 
actually other source nodes may have data avaiable. 

You've already gotten a number of suggestions, but I thought I'd add my
$0.02 to help you (hopefully) sort them out.

I agree with Terry that you should read up on MPI_ANY_SOURCE.  With
this, you can listen for any source process and use functions like
MPI_Recv or MPI_Irecv.  I further agree that you could consider
functions like MPI_Probe or MPI_Iprobe.

Win's examples don't seem to apply since they implement gathers (one
input from each process) and you explicitly said that no such
regularity can be assumed.

Biagio's suggestion might be helpful, but also sounds like it might be
overkill.

Re: [OMPI users] Relocating an Open MPI installation using OPAL_PREFIX

[Ethan Mallove]

On Thu, Dec/25/2008 08:12:49AM, Jeff Squyres wrote:
> It's quite possible that we don't handle this situation properly.  Won't 
> you need to libdir's (one for the 32 bit OMPI executables, and one for the 
> 64 bit MPI apps)?

I don't need an OPAL environment variable for the executables, just a
single OPAL_LIBDIR var for the libraries. (One set of 32-bit
executables runs with both 32-bit and 64-bit libraries.) I'm guessing
OPAL_LIBDIR will not work for you if you configure with a non-standard
--libdir option.

-Ethan


>
> On Dec 23, 2008, at 3:58 PM, Ethan Mallove wrote:
>
>> I think the problem is that I am doing a multi-lib build. I have
>> 32-bit libraries in lib/, and 64-bit libraries in lib/64. I assume I
>> do not see the issue for 32-bit tests, because all the dependencies
>> are where Open MPI expects them to be. For the 64-bit case, I tried
>> setting OPAL_LIBDIR to /opt/openmpi-relocated/lib/lib64, but no luck.
>> Given the below configure arguments, what do my OPAL_* env vars need
>> to be? (Also, could using --enable-orterun-prefix-by-default interfere
>> with OPAL_PREFIX?)
>>
>>$ ./configure CC=cc CXX=CC F77=f77 FC=f90  --with-openib 
>> --without-udapl --disable-openib-ibcm --enable-heterogeneous 
>> --enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by-default 
>> --with-sge --enable-mpi-f90 --with-mpi-f90-size=small 
>> --disable-mpi-threads --disable-progress-threads   --disable-debug  
>> CFLAGS="-m32 -xO5" CXXFLAGS="-m32 -xO5" FFLAGS="-m32 -xO5"  FCFLAGS="-m32 
>> -xO5" 
>> --prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install
>>  
>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/man
>>  
>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/lib
>>  
>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/include
>>  
>> --without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-install32 
>> --with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v!
>> 20z-12/ompi-tarball-testing/installs/DGQx/install 
>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/man
>>  
>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/lib
>>  
>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/include
>>  
>> LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/lib"
>>
>>$ ./confgiure CC=cc CXX=CC F77=f77 FC=f90  --with-openib 
>> --without-udapl --disable-openib-ibcm --enable-heterogeneous 
>> --enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by-default 
>> --with-sge --enable-mpi-f90 --with-mpi-f90-size=small 
>> --disable-mpi-threads --disable-progress-threads   --disable-debug  
>> CFLAGS="-m64 -xO5" CXXFLAGS="-m64 -xO5" FFLAGS="-m64 -xO5"  FCFLAGS="-m64 
>> -xO5" 
>> --prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install
>>  
>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/man
>>  
>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/lib/lib64
>>  
>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/include/64
>>  
>> --without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared-install64 
>> --with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt-scratch/!
>> burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install 
>> --mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/man
>>  
>> --libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/lib/lib64
>>  
>> --includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/include/64
>>  
>> LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install/lib"
>>  
>> --disable-binaries
>>
>> -Ethan
>>
>>>
>>>
>>> On Dec 22, 2008, at 12:42 PM, Ethan Mallove wrote:
>>>
 Can anyone get OPAL_PREFIX to work on Linux? A simple test is to see
 if the following works for any mpicc/mpirun:

 $ mv  /tmp/foo
 $ set OPAL_PREFIX /tmp/foo
 $ mpicc ...
 $ mpirun ...

 If you are able to get the above to run successfully, I'm interested
 in your config.log file.

 Thanks,
 Ethan


 On Thu, Dec/18/2008 11:03:25AM, Ethan Mallove wrote:
> Hello,
>
> The below FAQ lists instructions on how to use a relocated Open MPI
> installation:
>
> http://www.open-mpi.org/faq/?category=building#instal

Re: [OMPI users] mpirun hangs

[Maciej Kazulak]

2009/1/3 Maciej Kazulak 

> Hi,
>
> I have a weird problem. After a fresh install mpirun refuses to work:
>
> box% ./hello
> Process 0 on box out of 1
> box% mpirun -np 1 ./hello
> # hangs here, no output, nothing at all; on another terminal:
> box% ps axl | egrep 'mpirun|orted'
> 0  1000 24162  7687  20   0  86704  2744 -  Sl+  pts/2  0:00 mpirun
> -np 1 ./hello
> 1  1000 24165 1  20   0  76016  2088 -  Ss   ?  0:00 orted
> --bootproxy 1 --name 0.0.1 --num_procs 2 --vpid_start 0 --nodename box
> --universe ncl@box:default-universe-24162 --nsreplica "0.0.0;tcp://
> 192.168.1.8:21500" --gprreplica "0.0.0;tcp://192.168.1.8:21500" --set-sid
> 0  1000 24171 23924  20   0   6020   732 pipe_w S+   pts/3  0:00 egrep
> mpirun|orted
>
> Is there some post-install configuration i forgot to do? I couldn't find
> anything useful in the faq nor the docs that come with the package.
> Following advice in this thread
> http://www.open-mpi.org/community/lists/users/2007/08/3845.php i tried
> --debug-daemons but no output whatsoever as above.
> Also tried MTT:
>
> box% cat developer.ini trivial.ini| ../client/mtt -
> alreadyinstalled_dir=/usr
> ompi:version:full:1.2.8
> *** WARNING: Test: cxx_hello, np=2, variant=1: TIMED OUT (failed)
> *** WARNING: Test: c_ring, np=2, variant=1: TIMED OUT (failed)
> *** WARNING: Test: cxx_ring, np=2, variant=1: TIMED OUT (failed)
> *** WARNING: Test: c_hello, np=2, variant=1: TIMED OUT (failed)
> *** WARNING: Test: c_ring, np=2, variant=1: TIMED OUT (failed)
> *** WARNING: Test: c_hello, np=2, variant=1: TIMED OUT (failed)
>
> MTT Results Summary
> hostname: box
> uname: Linux box 2.6.28-gentoo #2 SMP Thu Jan 1 15:27:59 CET 2009 x86_64
> Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz GenuineIntel GNU/Linux
> who am i:
> +-+-+--+--+--+--+
> | Phase   | Section | Pass | Fail | Time out | Skip |
> +-+-+--+--+--+--+
> | MPI install | my installation | 1| 0| 0| 0|
> | MPI install | my installation | 1| 0| 0| 0|
> | Test Build  | trivial | 1| 0| 0| 0|
> | Test Build  | trivial | 1| 0| 0| 0|
> | Test Run| trivial | 0| 0| 4| 0|
> | Test Run| trivial | 0| 0| 2| 0|
> +-+-+--+--+--+--+
>
> box% ompi_info
> Open MPI: 1.2.8
>Open MPI SVN revision: r19718
> Open RTE: 1.2.8
>Open RTE SVN revision: r19718
> OPAL: 1.2.8
>OPAL SVN revision: r19718
>   Prefix: /usr
>  Configured architecture: x86_64-pc-linux-gnu
>Configured by: root
>Configured on: Sat Jan  3 01:03:53 CET 2009
>   Configure host: box
> Built by: root
> Built on: sob, 3 sty 2009, 01:06:54 CET
>   Built host: box
>   C bindings: yes
> C++ bindings: yes
>   Fortran77 bindings: no
>   Fortran90 bindings: no
>  Fortran90 bindings size: na
>   C compiler: x86_64-pc-linux-gnu-gcc
>  C compiler absolute: /usr/bin/x86_64-pc-linux-gnu-gcc
> C++ compiler: x86_64-pc-linux-gnu-g++
>C++ compiler absolute: /usr/bin/x86_64-pc-linux-gnu-g++
>   Fortran77 compiler: x86_64-pc-linux-gnu-gfortran
>   Fortran77 compiler abs: /usr/bin/x86_64-pc-linux-gnu-gfortran
>   Fortran90 compiler: none
>   Fortran90 compiler abs: none
>  C profiling: yes
>C++ profiling: yes
>  Fortran77 profiling: no
>  Fortran90 profiling: no
>   C++ exceptions: no
>   Thread support: posix (mpi: no, progress: no)
>   Internal debug support: no
>  MPI parameter check: runtime
> Memory profiling support: no
> Memory debugging support: no
>  libltdl support: yes
>Heterogeneous support: yes
>  mpirun default --prefix: yes
>MCA backtrace: execinfo (MCA v1.0, API v1.0, Component v1.2.8)
>   MCA memory: ptmalloc2 (MCA v1.0, API v1.0, Component v1.2.8)
>MCA paffinity: linux (MCA v1.0, API v1.0, Component v1.2.8)
>MCA maffinity: first_use (MCA v1.0, API v1.0, Component v1.2.8)
>MCA timer: linux (MCA v1.0, API v1.0, Component v1.2.8)
>  MCA installdirs: env (MCA v1.0, API v1.0, Component v1.2.8)
>  MCA installdirs: config (MCA v1.0, API v1.0, Component v1.2.8)
>MCA allocator: basic (MCA v1.0, API v1.0, Component v1.0)
>MCA allocator: bucket (MCA v1.0, API v1.0, Component v1.0)
> MCA coll: basic (MCA v1.0, API v1.0, Component v1.2.8)
> MCA coll: self (MCA v1.0, API v1.0, Component v1.2.8)
> MCA coll: sm (MCA v1.0, API v1.0, Component v1.2.8)
> MCA coll: tuned (MCA v1.0, API v1.0, Component v1.2.8)
> 

Re: [OMPI users] using ompi-server on a single node

[Terry Frankcombe]

But why doesn't tcp work on loopback?


On Mon, 2009-01-05 at 07:25 -0700, Ralph Castain wrote:
> It is currently a known limitation - shared memory currently only  
> works between procs from the same job. There is an enhancement coming  
> that will remove this restriction, but it won't be out for some time.
> 
> Ralph
> 
> On Jan 5, 2009, at 1:06 AM, Thomas Ropars wrote:
> 
> > Hi,
> >
> > I've tried to use ompi-server to connect 2 processes belonging to
> > different jobs but running on the same computer. It works when the
> > computer has a network interface up. But if the only active network
> > interface is the local loop, it doesn't work.
> >
> > According to what I understood reading the code, it is because no btl
> > component can be used in this case. "tcp" is not used because usually
> > it is the "sm" component that is used for processes on the same host.
> > But in that case it doesn't work because "sm" is supposed to work only
> > for processes of the same job.
> >
> > I know that this use-case is not very frequent  :)
> > But Is there a solution to make it work ? or is it a known  
> > limitation ?
> >
> > Regards
> >
> > Thomas
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] mpirun hangs

[Jeff Squyres]

On Jan 5, 2009, at 5:01 PM, Maciej Kazulak wrote:

Interesting though. I thought in such a simple scenario shared  
memory would be used for IPC (or whatever's fastest) . But nope.  
Even with one process still it wants to use TCP/IP to communicate  
between mpirun and orted.


Correct -- we only have TCP enabled for MPI process <--> orted  
communication.  There are several reasons why; the simplest is that  
this is our "out of band" channel and it is only used to setup and  
tear down the job.  As such, we don't care that it's a little slower  
than other possible channels (such as sm).  MPI traffic will use  
shmem, OpenFabrics-based networks, Myrinet, ...etc.  But not MPI  
process <--> orted communication.


What's even more surprising to me it won't use loopback for that.  
Hence my maybe a little bit over-restrictive iptables rules were the  
problem. I allowed traffic from 127.0.0.1 to 127.0.0.1 on lo but not  
from  to  on eth0 and both processes ended up  
waiting for IO.


Can I somehow configure it to use something other than TCP/IP here?  
Or at least switch it to loopback?


I don't remember how it works in the v1.2 series offhand; I think it's  
different in the v1.3 series (where all MPI processes *only* talk to  
the local orted, vs. MPI processes making direct TCP connections back  
to mpirun and any other MPI process with which it needs to bootstrap  
other communication channels).  I'm *guessing* that the MPI process <-- 
> orted communication either uses a named unix socket or TCP  
loopback.  Ralph -- can you explain the details?


--
Jeff Squyres
Cisco Systems

Re: [OMPI users] Relocating an Open MPI installation using OPAL_PREFIX

[Jeff Squyres]

I honestly haven't thought through the ramifications of doing a multi- 
lib build with OPAL_PREFIX et al. :-\


If you setenv OPAL_LIBDIR, it'll use whatever you set it to, so it  
doesn't matter what you configured --libdir with.  Additionally mca/ 
installdirs/config/install_dirs.h has this by default:


#define OPAL_LIBDIR "${exec_prefix}/lib"

Hence, if you use a default --libdir and setenv OPAL_PREFIX, then the  
libdir should pick up the right thing (because it's based on the  
prefix).  But if you use --libdir that is *not* based on $ 
{exec_prefix}, then you might run into problems.


Perhaps you can '--libdir="${exec_prefix}/lib64"' so that you can have  
your custom libdir, but still have it dependent upon the prefix that  
gets expanded at run time...?


(again, I'm not thinking all of this through -- just offering a few  
suggestions off the top of my head that you'll need to test / trace  
the code to be sure...)



On Jan 5, 2009, at 1:35 PM, Ethan Mallove wrote:


On Thu, Dec/25/2008 08:12:49AM, Jeff Squyres wrote:
It's quite possible that we don't handle this situation properly.   
Won't
you need to libdir's (one for the 32 bit OMPI executables, and one  
for the

64 bit MPI apps)?


I don't need an OPAL environment variable for the executables, just a
single OPAL_LIBDIR var for the libraries. (One set of 32-bit
executables runs with both 32-bit and 64-bit libraries.) I'm guessing
OPAL_LIBDIR will not work for you if you configure with a non-standard
--libdir option.

-Ethan




On Dec 23, 2008, at 3:58 PM, Ethan Mallove wrote:


I think the problem is that I am doing a multi-lib build. I have
32-bit libraries in lib/, and 64-bit libraries in lib/64. I assume I
do not see the issue for 32-bit tests, because all the dependencies
are where Open MPI expects them to be. For the 64-bit case, I tried
setting OPAL_LIBDIR to /opt/openmpi-relocated/lib/lib64, but no  
luck.

Given the below configure arguments, what do my OPAL_* env vars need
to be? (Also, could using --enable-orterun-prefix-by-default  
interfere

with OPAL_PREFIX?)

  $ ./configure CC=cc CXX=CC F77=f77 FC=f90  --with-openib
--without-udapl --disable-openib-ibcm --enable-heterogeneous
--enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by- 
default

--with-sge --enable-mpi-f90 --with-mpi-f90-size=small
--disable-mpi-threads --disable-progress-threads   --disable-debug
CFLAGS="-m32 -xO5" CXXFLAGS="-m32 -xO5" FFLAGS="-m32 -xO5"   
FCFLAGS="-m32

-xO5"
--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/lib
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ 
ompi-tarball-testing/installs/DGQx/install/include
--without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared- 
install32
--with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt- 
scratch/burl-ct-v!

20z-12/ompi-tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/lib
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ 
ompi-tarball-testing/installs/DGQx/install/include
LDFLAGS=-R/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/lib"


  $ ./confgiure CC=cc CXX=CC F77=f77 FC=f90  --with-openib
--without-udapl --disable-openib-ibcm --enable-heterogeneous
--enable-cxx-exceptions --enable-shared --enable-orterun-prefix-by- 
default

--with-sge --enable-mpi-f90 --with-mpi-f90-size=small
--disable-mpi-threads --disable-progress-threads   --disable-debug
CFLAGS="-m64 -xO5" CXXFLAGS="-m64 -xO5" FFLAGS="-m64 -xO5"   
FCFLAGS="-m64

-xO5"
--prefix=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/lib/lib64
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ 
ompi-tarball-testing/installs/DGQx/install/include/64
--without-mx --with-tm=/ws/ompi-tools/orte/torque/current/shared- 
install64
--with-contrib-vt-flags="--prefix=/workspace/em162155/hpc/mtt- 
scratch/!

burl-ct-v20z-12/ompi-tarball-testing/installs/DGQx/install
--mandir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/man
--libdir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ompi- 
tarball-testing/installs/DGQx/install/lib/lib64
--includedir=/workspace/em162155/hpc/mtt-scratch/burl-ct-v20z-12/ 
ompi-tarball-testing/installs/DGQx/install/include/64