$ /usr/bin/gcc --version Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix
The other thing that can go wrong is the configuration of the interfaces that allow different pro cesses to communicate together. Let’s se what the default is on your machine. Can you report the output of the two following commands: ompi_info —param oob tcp —level 9 ompi_info —param btl tcp —level 9 Another thing you can try is to manually set the interface to be used. I would try lo0, en0 or en1 (if available). You can set oob_tcp_if_include and btl_tcp_if_include to one of these values to see what is happening. George. On Nov 27, 2013, at 19:55 , Meredith, Karl <karl.mered...@fmglobal.com> wrote: > So, I decided to try out your configuration flags to see if that’d work. All > Apple tools, no macports. > > With openmpi-1.7.3 I did the following: > > make clean > ./configure --prefix=/opt/trunk/apple-only --enable-shared --disable-static > --enable-debug --disable-io-romio --enable-contrib-no-build=vt,libtrace > --enable-mpirun-prefix-by-default > make > make install > > export PATH=/opt/trunk/apple-only/bin/:$PATH > export LD_LIBRARY_PATH=/opt/trunk/apple-only/lib/:$LD_LIBRARY_PATH > export DYLD_LIBRARY_PATH=/opt/trunk/apple-only/lib/:$DYLD_LIBRARY_PATH > > bash-4.2$ echo $PATH > /opt/trunk/apple-only/bin/:/usr/bin:/bin:/usr/sbin:/sbin > bash-4.2$ echo $LD_LIBRARY_PATH > /opt/trunk/apple-only/lib/: > bash-4.2$ echo $DYLD_LIBRARY_PATH > /opt/trunk/apple-only/lib/: > > cd /Users/meredithk/tools/openmpi-1.7.3-src/examples > bash-4.2$ make clean > rm -f hello_c hello_cxx hello_mpifh hello_usempi hello_usempif08 Hello.class > ring_c ring_cxx ring_mpifh ring_usempi ring_usempif08 Ring.class > connectivity_c *~ *.o > bash-4.2$ make > mpicc -g hello_c.c -o hello_c > mpicc -g ring_c.c -o ring_c > mpicc -g connectivity_c.c -o connectivity_c > mpic++ -g hello_cxx.cc -o hello_cxx > mpic++ -g ring_cxx.cc -o ring_cxx > > bash-4.2$ mpirun -np 2 ./hello_c > (No output, program stalls.) > > > Is your version of clang the same as mine? Apple LLVM version 5.0 > (clang-500.2.79) (based on LLVM 3.3svn) > > Are you using openmpi-1.7.3? > > Karl > > On Nov 27, 2013, at 1:26 PM, George Bosilca <bosi...@icl.utk.edu> wrote: > >> Puzzling! I removed my entire Mac Ports installation in order to fall back >> to the default set of tools provided by Apple via the command tool package. >> I recompile a fresh copy of Open MPI with the following configure: >> >> ./configure —prefix=/opt/trunk/apple-only —enable-shared —disable-static >> --enable-debug --disable-io-romio --enable-contrib-no-build=vt,libtrace >> --enable-mpirun-prefix-by-default >> >> And every test I throw at it passed without any issue. Totally puzzling … >> >> George. >> >> >> On Nov 27, 2013, at 18:04 , Meredith, Karl <karl.mered...@fmglobal.com> >> wrote: >> >>> None of the C or C++ examples run. For example, the hello_c.c example in >>> the examples directory included with openmpi. It compiles, but hangs on >>> MPI_Init(). >>> >>> Are your compiler versions different from the ones I’m showing? Apple LLVM >>> version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) >>> >>> Karl >>> >>> >>> On Nov 27, 2013, at 11:51 AM, Nathan Hjelm <hje...@lanl.gov> wrote: >>> >>>> Been running on Mavericks all summer with no issues. I do not use the C++ >>>> interface >>>> though (C++ bindings were removed from the MPI standard in 3.0.) Can you >>>> try a C >>>> example and see if that works? >>>> >>>> -Nathan Hjelm >>>> Application Readiness, HPC-5, LANL >>>> >>>> On Wed, Nov 27, 2013 at 11:43:05AM -0500, Meredith, Karl wrot >>>>> Okay, in order to try and track down this problem I have done a fresh >>>>> install of OpenMPI-1.7.3 on Mac OS 10.9 (Mavericks). I am using the >>>>> Apple compilers, and not using anything from macports. The code compiles >>>>> fine, but when running the examples for openmpi-1.7.3, the code hangs in >>>>> MPI::Init() and never returns. No error message. No warning. It just >>>>> hangs. If I compile in debug mode and do a stack trace, it shows that it >>>>> is stalled inside of MPI::Init() every time. Has anyone been able to >>>>> successfully get openmpi working on 10.9? If so, what, if anything, are >>>>> you doing differently? >>>>> >>>>> This is how I installed openmpi: >>>>> ./configure --prefix=$HOME/tools/openmpi-1.7.3 >>>>> make >>>>> make install >>>>> >>>>> Here are my settings, environment variables, compilers, etc: >>>>> >>>>> bash-4.2$ which g++ >>>>> /usr/bin/g++ >>>>> >>>>> bash-4.2$ which gcc >>>>> /usr/bin/gcc >>>>> >>>>> bash-4.2$ which clang >>>>> /usr/bin/clang >>>>> >>>>> bash-4.2$ g++ --version >>>>> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr >>>>> --with-gxx-include-dir=/usr/include/c++/4.2.1 >>>>> Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) >>>>> Target: x86_64-apple-darwin13.0.0 >>>>> Thread model: posix >>>>> >>>>> bash-4.2$ gcc --version >>>>> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr >>>>> --with-gxx-include-dir=/usr/include/c++/4.2.1 >>>>> Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) >>>>> Target: x86_64-apple-darwin13.0.0 >>>>> Thread model: posix >>>>> >>>>> bash-4.2$ which mpicc >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/bin/mpicc >>>>> >>>>> bash-4.2$ which mpic++ >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/bin/mpic++ >>>>> >>>>> bash-4.2$ which mpirun >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/bin/mpirun >>>>> >>>>> bash-4.2$ mpicc -show >>>>> gcc -I/Users/meredithk/tools/openmpi-1.7.3-noenv/include >>>>> -L/Users/meredithk/tools/openmpi-1.7.3-noenv/lib -lmpi >>>>> >>>>> bash-4.2$ mpic++ -show >>>>> g++ -I/Users/meredithk/tools/openmpi-1.7.3-noenv/include >>>>> -L/Users/meredithk/tools/openmpi-1.7.3-noenv/lib -lmpi_cxx -lmpi >>>>> >>>>> bash-4.2$ pwd >>>>> /Users/meredithk/tools/openmpi-1.7.3-src >>>>> >>>>> bash-4.2$ otool -L hello_cxx >>>>> hello_cxx: >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libmpi_cxx.1.dylib >>>>> (compatibility version 3.0.0, current version 3.3.0) >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libmpi.1.dylib >>>>> (compatibility version 4.0.0, current version 4.0.0) >>>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version >>>>> 120.0.0) >>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version >>>>> 1197.1.1) >>>>> >>>>> bash-4.2$ mpirun -n 5 hostname >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com> >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com> >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com> >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com> >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com> >>>>> >>>>> bash-4.2$ echo $PATH >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/bin:/usr/bin:/bin:/usr/sbin:/sbin >>>>> >>>>> bash-4.2$ echo $LD_LIBRARY_PATH >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib: >>>>> >>>>> bash-4.2$ echo $DYLD_LIBRARY_PATH >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib: >>>>> >>>>> bash-4.2$ ls /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/ >>>>> libmca_common_sm.4.dylib libmpi_cxx.1.dylib libopen-pal.6.dylib >>>>> libopen-trace-format.1.dylib libotfaux.dylib libvt-mpi-unify.0.dylib >>>>> libvt-mpi.dylib libvt.0.dylib >>>>> libmca_common_sm.dylib libmpi_cxx.dylib libopen-pal.dylib >>>>> libopen-trace-format.a libotfaux.la libvt-mpi-unify.a libvt-mpi.la libvt.a >>>>> libmca_common_sm.la libmpi_cxx.la libopen-pal.la >>>>> libopen-trace-format.dylib libvt-hyb.0.dylib libvt-mpi-unify.dylib >>>>> libvt-mt.0.dylib libvt.dylib >>>>> libmpi.1.dylib libompitrace.0.dylib libopen-rte.6.dylib >>>>> libopen-trace-format.la libvt-hyb.a libvt-mpi-unify.la libvt-mt.a libvt.la >>>>> libmpi.dylib libompitrace.dylib libopen-rte.dylib libotfaux.0.dylib >>>>> libvt-hyb.dylib libvt-mpi.0.dylib libvt-mt.dylib openmpi >>>>> libmpi.la libompitrace.la libopen-rte.la libotfaux.a libvt-hyb.la >>>>> libvt-mpi.a libvt-mt.la pkgconfig >>>>> >>>>> bash-4.2$ otool -L >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libmpi_cxx.dylib >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libmpi_cxx.dylib: >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libmpi_cxx.1.dylib >>>>> (compatibility version 3.0.0, current version 3.3.0) >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libmpi.1.dylib >>>>> (compatibility version 4.0.0, current version 4.0.0) >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libopen-rte.6.dylib >>>>> (compatibility version 7.0.0, current version 7.0.0) >>>>> /Users/meredithk/tools/openmpi-1.7.3-noenv/lib/libopen-pal.6.dylib >>>>> (compatibility version 7.0.0, current version 7.0.0) >>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version >>>>> 1197.1.1) >>>>> /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version >>>>> 120.0.0) >>>>> >>>>> On Nov 27, 2013, at 9:41 AM, Meredith, Karl >>>>> <karl.mered...@fmglobal.com<mailto:karl.mered...@fmglobal.com>> wrote: >>>>> >>>>> I’m pretty sure it is using the correct g++ >>>>> >>>>> $ )which g++ >>>>> /opt/local/bin/g++ >>>>> >>>>> >>>>> $ )echo $PATH >>>>> /Users/meredithk/tools/openmpi/bin:/opt/local/bin:/opt/local/sbin:/Users/meredithk/tools/bin:/Users/meredithk/OpenFOAM/fireFoam-2.2.x/scripts:/Users/meredithk/OpenFOAM/ThirdParty-2.2.x/platforms/darwinIntel64Gcc46/gperftools-svn/bin:/Applications/paraview.app/Contents/bin:/Users/meredithk/OpenFOAM/ThirdParty-2.2.x/platforms/darwinIntel64Gcc46/paraview-3.12.0/bin:/Users/meredithk/OpenFOAM/meredithk-2.2.x/platforms/darwinIntel64Gcc46DPOpt/bin:/Users/meredithk/OpenFOAM/site/2.2.x/platforms/darwinIntel64Gcc46DPOpt/bin:/Users/meredithk/OpenFOAM/OpenFOAM-2.2.x/platforms/darwinIntel64Gcc46DPOpt/bin:/Users/meredithk/OpenFOAM/OpenFOAM-2.2.x/bin:/Users/meredithk/OpenFOAM/OpenFOAM-2.2.x/wmake:/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin >>>>> >>>>> >>>>> $ )g++ --version >>>>> g++ (MacPorts gcc46 4.6.4_3) 4.6.4 >>>>> >>>>> >>>>> Karl >>>>> >>>>> >>>>> >>>>> On Nov 27, 2013, at 3:23 AM, Iliev, Hristo >>>>> <il...@rz.rwth-aachen.de<mailto:il...@rz.rwth-aachen.de><mailto:il...@rz.rwth-aachen.de>> >>>>> wrote: >>>>> >>>>> Karl, >>>>> >>>>> The Xcode command-line tools provide a g++ command that is effectively >>>>> clang++. Since mpic++ invokes g++ without the full path to it, it might >>>>> pick >>>>> the wrong g++. Make sure that /opt/local/bin is the first item in your >>>>> $PATH. >>>>> >>>>> Hristo >>>>> >>>>> -- >>>>> Hristo Iliev, PhD - High Performance Computing Team / JARA-HPC >>>>> RWTH Aachen University, Center for Computing and Communication >>>>> Rechen- und Kommunikationszentrum der RWTH Aachen >>>>> Seffenter Weg 23, D-52074 Aachen (Germany) >>>>> >>>>> -----Original Message----- >>>>> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Meredith, >>>>> Karl >>>>> Sent: Tuesday, November 26, 2013 9:33 PM >>>>> To: Open MPI Users >>>>> Subject: Re: [OMPI users] open-mpi on Mac OS 10.9 (Mavericks) >>>>> >>>>> Here are the results of those two commands: >>>>> >>>>> $ )mpic++ -show >>>>> g++ -I/Users/meredithk/tools/openmpi/include - >>>>> L/Users/meredithk/tools/openmpi/lib -lmpi_cxx -lmpi -lm >>>>> >>>>> $ )otool -L /Users/meredithk/tools/openmpi/lib/libmpi_cxx.dylib >>>>> /Users/meredithk/tools/openmpi/lib/libmpi_cxx.dylib: >>>>> /Users/meredithk/tools/openmpi/lib/libmpi_cxx.1.dylib >>>>> (compatibility version 2.0.0, current version 2.2.0) >>>>> /Users/meredithk/tools/openmpi/lib/libmpi.1.dylib (compatibility >>>>> version 2.0.0, current version 2.8.0) >>>>> /opt/local/lib/libgcc/libstdc++.6.dylib (compatibility version >>>>> 7.0.0, >>>>> current version 7.18.0) >>>>> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >>>>> version >>>>> 1197.1.1) >>>>> /opt/local/lib/libgcc/libgcc_s.1.dylib (compatibility version 1.0.0, >>>>> current version 1.0.0) >>>>> >>>>> It looks like libstdc++ is being used and g++ is being used. >>>>> >>>>> Karl >>>>> >>>>> >>>>> On Nov 25, 2013, at 6:56 PM, Pierre Jolivet >>>>> <joli...@ann.jussieu.fr<mailto:joli...@ann.jussieu.fr><mailto:joli...@ann.jussieu.fr>> >>>>> wrote: >>>>> >>>>> Karl, >>>>> What does "mpic++ -show" returns ? It is possible that you are >>>>> compiling/linking with "c++", which will defaults to clang++, while you >>>>> compiled OpenMPI with g++. >>>>> Since libstdc++ and libc++ have incompatible ABI, that might be why you >>>>> are getting a wrong behavior. >>>>> Also, it could be worthwhile to have a look at "otool -L >>>>> /Users/meredithk/tools/openmpi/lib/libmpi_cxx.dylib" >>>>> Pierre >>>>> >>>>> On Nov 25, 2013, at 7:38 AM, Meredith, Karl >>>>> <karl.mered...@fmglobal.com<mailto:karl.mered...@fmglobal.com><mailto:karl.mered...@fmglobal.com>> >>>>> wrote: >>>>> >>>>> I do have DYLD_LIBRARY_PATH set as well, and I get the same problem: >>>>> >>>>> DYLD_LIBRARY_PATH=/Users/meredithk/tools/openmpi/lib >>>>> >>>>> Here's the directory listing of /Users/meredithk/tools/openmpi/lib >>>>> >>>>> $ )ls -1 >>>>> libmca_common_sm.3.dylib* >>>>> libmca_common_sm.dylib@ >>>>> libmca_common_sm.la* >>>>> libmpi.1.dylib* >>>>> libmpi.dylib@ >>>>> libmpi.la* >>>>> libmpi_cxx.1.dylib* >>>>> libmpi_cxx.dylib@ >>>>> libmpi_cxx.la* >>>>> libmpi_f77.1.dylib* >>>>> libmpi_f77.dylib@ >>>>> libmpi_f77.la* >>>>> libmpi_f90.a >>>>> libmpi_f90.la* >>>>> libompitrace.0.dylib* >>>>> libompitrace.dylib@ >>>>> libompitrace.la* >>>>> libopen-pal.4.dylib* >>>>> libopen-pal.dylib@ >>>>> libopen-pal.la* >>>>> libopen-rte.4.dylib* >>>>> libopen-rte.dylib@ >>>>> libopen-rte.la* >>>>> libopen-trace-format.1.dylib* >>>>> libopen-trace-format.a >>>>> libopen-trace-format.dylib@ >>>>> libopen-trace-format.la* >>>>> libotfaux.0.dylib* >>>>> libotfaux.a >>>>> libotfaux.dylib@ >>>>> libotfaux.la* >>>>> libvt-hyb.0.dylib* >>>>> libvt-hyb.a >>>>> libvt-hyb.dylib@ >>>>> libvt-hyb.la* >>>>> libvt-mpi-unify.0.dylib* >>>>> libvt-mpi-unify.a >>>>> libvt-mpi-unify.dylib@ >>>>> libvt-mpi-unify.la* >>>>> libvt-mpi.0.dylib* >>>>> libvt-mpi.a >>>>> libvt-mpi.dylib@ >>>>> libvt-mpi.la* >>>>> libvt-mt.0.dylib* >>>>> libvt-mt.a >>>>> libvt-mt.dylib@ >>>>> libvt-mt.la* >>>>> libvt-pomp.a >>>>> libvt-pomp.la* >>>>> libvt.0.dylib* >>>>> libvt.a >>>>> libvt.dylib@ >>>>> libvt.la* >>>>> mpi.mod* >>>>> openmpi/ >>>>> pkgconfig/ >>>>> >>>>> >>>>> I don't notice anything out of ordinary. >>>>> >>>>> Karl >>>>> >>>>> >>>>> On Nov 25, 2013, at 8:31 AM, Reuti >>>>> <re...@staff.uni-marburg.de<mailto:re...@staff.uni-marburg.de><mailto:re...@staff.uni-marburg.de>> >>>>> wrote: >>>>> >>>>> Am 25.11.2013 um 14:25 schrieb Meredith, Karl: >>>>> >>>>> I do have these two environment variables set: >>>>> >>>>> LD_LIBRARY_PATH=/Users/meredithk/tools/openmpi/lib >>>>> >>>>> On a Mac it should DYLD_LIBRARY_PATH - and there are *.dylib files in >>>>> your /Users/meredithk/tools/openmpi/lib? >>>>> >>>>> -- Reuti >>>>> >>>>> >>>>> PATH=/Users/meredithk/tools/openmpi/bin >>>>> >>>>> Running mpirun seems to work fine with a simple command, like >>>>> hostname: >>>>> >>>>> $ )mpirun -n 2 hostname >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com/><http://meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com/>> >>>>> meredithk-mac.corp.fmglobal.com<http://meredithk-mac.corp.fmglobal.com/> >>>>> >>>>> I am trying to run the simple hello_cxx example from the openmpi >>>>> distribution, compiled as such: >>>>> mpic++ -g hello_cxx.cc -o hello_cxx >>>>> >>>>> It compiles fine, without warning or error. However, when I go to >>>>> run >>>>> the example, it stalls on the MPI::Init() command: >>>>> mpirun -np 1 hello_cxx >>>>> It never errors out or crashes. It simply hangs. >>>>> >>>>> I am using the same mpic++ and mpirun version: >>>>> $ )which mpirun >>>>> /Users/meredithk/tools/openmpi/bin/mpirun >>>>> >>>>> $ )which mpic++ >>>>> /Users/meredithk/tools/openmpi/bin/mpic++ >>>>> >>>>> Not quite sure what else to check. >>>>> >>>>> Karl >>>>> >>>>> >>>>> On Nov 23, 2013, at 5:29 PM, Ralph Castain >>>>> <r...@open-mpi.org<mailto:r...@open-mpi.org><mailto:r...@open-mpi.org>> >>>>> wrote: >>>>> >>>>> Strange - I run on Mavericks now without problem. Can you run >>>>> "mpirun -n 1 hostname"? >>>>> >>>>> You also might want to check your PATH and LD_LIBRARY_PATH to >>>>> ensure you have the prefix where you installed OMPI 1.6.5 at the front. >>>>> Mac >>>>> distributes a very old version of OMPI with its software and you don't >>>>> want >>>>> to pick it up by mistake. >>>>> >>>>> >>>>> On Nov 22, 2013, at 1:45 PM, Meredith, Karl >>>>> <karl.mered...@fmglobal.com<mailto:karl.mered...@fmglobal.com><mailto:karl.mered...@fmglobal.com>> >>>>> wrote: >>>>> >>>>> I recently upgraded my 2013 Macbook Pro (Retina display) from 10.8 >>>>> to 10.9. I downloaded and installed openmpi-1.6.5 and compiled it with >>>>> gcc >>>>> 4.8 (gcc installed from macports). >>>>> openmpi compiled and installed without error. >>>>> >>>>> However, when I try to run any of the example test cases, the code >>>>> gets stuck inside the first MPI::Init() call and never returns. >>>>> >>>>> Any thoughts on what might be going wrong? >>>>> >>>>> The same install on OS 10.8 works fine and the example test cases >>>>> run without error. >>>>> >>>>> Karl >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> <smime.p7s>_______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org><mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org<mailto:us...@open-mpi.org> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> <ATT00001>_______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
