[OMPI users] undefined reference to `MPI::Comm::Comm()

[Tomek Wlodarski]

Hi,

I am trying to locally compile software which uses openmpi (1.6.3),
but I got this error:

restraint_camshift2.o:(.toc+0x98): undefined reference to
`ompi_mpi_cxx_op_intercept'
restraint_camshift2.o: In function `Intracomm':
/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
undefined reference to `MPI::Comm::Comm()'
/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
undefined reference to `MPI::Comm::Comm()'
restraint_camshift2.o: In function `Intracomm':
/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:
undefined reference to `MPI::Comm::Comm()'
restraint_camshift2.o: In function `Intracomm':
/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
undefined reference to `MPI::Comm::Comm()'
/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
undefined reference to `MPI::Comm::Comm()'
restraint_camshift2.o:/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
more undefined references to `MPI::Comm::Comm()' follow
restraint_camshift2.o:(.data.rel.ro._ZTVN3MPI3WinE[_ZTVN3MPI3WinE]+0x48):
undefined reference to `MPI::Win::Free()'
restraint_camshift2.o:(.data.rel.ro._ZTVN3MPI8DatatypeE[_ZTVN3MPI8DatatypeE]+0x78):
undefined reference to `MPI::Datatype::Free()'
collect2: error: ld returned 1 exit status
make[3]: *** [mdrun] Error 1
make[3]: Leaving directory `/home/users/didymos/src/gromacs-4.5.5/src/kernel'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/home/users/didymos/src/gromacs-4.5.5/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/users/didymos/src/gromacs-4.5.5/src'
make: *** [all-recursive] Error 1

I am using gcc 4.7.3
Any ideas or suggestions?
Thanks!
Best,

tomek

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Jeff Squyres (jsquyres)]

Please send all the information listed here:

http://www.open-mpi.org/community/help/


On Jul 9, 2013, at 8:36 AM, Tomek Wlodarski  wrote:

> Hi,
> 
> I am trying to locally compile software which uses openmpi (1.6.3),
> but I got this error:
> 
> restraint_camshift2.o:(.toc+0x98): undefined reference to
> `ompi_mpi_cxx_op_intercept'
> restraint_camshift2.o: In function `Intracomm':
> /home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
> undefined reference to `MPI::Comm::Comm()'
> /home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
> undefined reference to `MPI::Comm::Comm()'
> restraint_camshift2.o: In function `Intracomm':
> /home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm_inln.h:23:
> undefined reference to `MPI::Comm::Comm()'
> restraint_camshift2.o: In function `Intracomm':
> /home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
> undefined reference to `MPI::Comm::Comm()'
> /home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
> undefined reference to `MPI::Comm::Comm()'
> restraint_camshift2.o:/home/users/didymos/openmpi-1.6.3/include/openmpi/ompi/mpi/cxx/intracomm.h:25:
> more undefined references to `MPI::Comm::Comm()' follow
> restraint_camshift2.o:(.data.rel.ro._ZTVN3MPI3WinE[_ZTVN3MPI3WinE]+0x48):
> undefined reference to `MPI::Win::Free()'
> restraint_camshift2.o:(.data.rel.ro._ZTVN3MPI8DatatypeE[_ZTVN3MPI8DatatypeE]+0x78):
> undefined reference to `MPI::Datatype::Free()'
> collect2: error: ld returned 1 exit status
> make[3]: *** [mdrun] Error 1
> make[3]: Leaving directory `/home/users/didymos/src/gromacs-4.5.5/src/kernel'
> make[2]: *** [all-recursive] Error 1
> make[2]: Leaving directory `/home/users/didymos/src/gromacs-4.5.5/src'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/home/users/didymos/src/gromacs-4.5.5/src'
> make: *** [all-recursive] Error 1
> 
> I am using gcc 4.7.3
> Any ideas or suggestions?
> Thanks!
> Best,
> 
> tomek
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Tomek Wlodarski]

So I am running OpenMPi1.6.3 (config.log attached)
And I would like to install gromacs patched with plumed (scientific
computing). Both uses openmpi.
Gromacs alone compiles without errors (openMPI works). But when
patched I got one mentioned before.
I am sending config file for patched gromacs.
If you need any other file I would be happy to provide.
Thanks a lot!
Best,

tomek


config_gromacs.log.bz2
Description: BZip2 compressed data


config_openmpi.log.bz2
Description: BZip2 compressed data

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Jeff Squyres (jsquyres)]

I don't see all the info requested from that web page, but it looks like OMPI 
built the C++ bindings ok.

Did you use mpic++ to build Gromacs?


On Jul 9, 2013, at 9:20 AM, Tomek Wlodarski  wrote:

> So I am running OpenMPi1.6.3 (config.log attached)
> And I would like to install gromacs patched with plumed (scientific
> computing). Both uses openmpi.
> Gromacs alone compiles without errors (openMPI works). But when
> patched I got one mentioned before.
> I am sending config file for patched gromacs.
> If you need any other file I would be happy to provide.
> Thanks a lot!
> Best,
> 
> tomek
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Tomek Wlodarski]

I used mpicc but when I switched in Makefile to mpic++ it compiled
without errors.
Thanks a lot!
Best,

tomek

On Tue, Jul 9, 2013 at 2:31 PM, Jeff Squyres (jsquyres)
 wrote:
> I don't see all the info requested from that web page, but it looks like OMPI 
> built the C++ bindings ok.
>
> Did you use mpic++ to build Gromacs?
>
>
> On Jul 9, 2013, at 9:20 AM, Tomek Wlodarski  wrote:
>
>> So I am running OpenMPi1.6.3 (config.log attached)
>> And I would like to install gromacs patched with plumed (scientific
>> computing). Both uses openmpi.
>> Gromacs alone compiles without errors (openMPI works). But when
>> patched I got one mentioned before.
>> I am sending config file for patched gromacs.
>> If you need any other file I would be happy to provide.
>> Thanks a lot!
>> Best,
>>
>> tomek
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Jeff Squyres (jsquyres)]

If you care, the issue is that it looks like Gromacs is using the MPI C++ 
bindings.  You therefore need to use the MPI C++ wrapper compiler, mpic++ (vs. 
mpicc, which is the MPI C wrapper compiler).


On Jul 9, 2013, at 9:41 AM, Tomek Wlodarski  wrote:

> I used mpicc but when I switched in Makefile to mpic++ it compiled
> without errors.
> Thanks a lot!
> Best,
> 
> tomek
> 
> On Tue, Jul 9, 2013 at 2:31 PM, Jeff Squyres (jsquyres)
>  wrote:
>> I don't see all the info requested from that web page, but it looks like 
>> OMPI built the C++ bindings ok.
>> 
>> Did you use mpic++ to build Gromacs?
>> 
>> 
>> On Jul 9, 2013, at 9:20 AM, Tomek Wlodarski  
>> wrote:
>> 
>>> So I am running OpenMPi1.6.3 (config.log attached)
>>> And I would like to install gromacs patched with plumed (scientific
>>> computing). Both uses openmpi.
>>> Gromacs alone compiles without errors (openMPI works). But when
>>> patched I got one mentioned before.
>>> I am sending config file for patched gromacs.
>>> If you need any other file I would be happy to provide.
>>> Thanks a lot!
>>> Best,
>>> 
>>> tomek
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> --
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Tomek Wlodarski]

Oh you are right.
Thanks.
Best

tomek

On Tue, Jul 9, 2013 at 2:44 PM, Jeff Squyres (jsquyres)
 wrote:
> If you care, the issue is that it looks like Gromacs is using the MPI C++ 
> bindings.  You therefore need to use the MPI C++ wrapper compiler, mpic++ 
> (vs. mpicc, which is the MPI C wrapper compiler).
>
>
> On Jul 9, 2013, at 9:41 AM, Tomek Wlodarski  wrote:
>
>> I used mpicc but when I switched in Makefile to mpic++ it compiled
>> without errors.
>> Thanks a lot!
>> Best,
>>
>> tomek
>>
>> On Tue, Jul 9, 2013 at 2:31 PM, Jeff Squyres (jsquyres)
>>  wrote:
>>> I don't see all the info requested from that web page, but it looks like 
>>> OMPI built the C++ bindings ok.
>>>
>>> Did you use mpic++ to build Gromacs?
>>>
>>>
>>> On Jul 9, 2013, at 9:20 AM, Tomek Wlodarski  
>>> wrote:
>>>
 So I am running OpenMPi1.6.3 (config.log attached)
 And I would like to install gromacs patched with plumed (scientific
 computing). Both uses openmpi.
 Gromacs alone compiles without errors (openMPI works). But when
 patched I got one mentioned before.
 I am sending config file for patched gromacs.
 If you need any other file I would be happy to provide.
 Thanks a lot!
 Best,

 tomek
 ___
 users mailing list
 us...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to: 
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] undefined reference to `MPI::Comm::Comm()

[Jeff Hammond]

You must be using an older version of Gromacs, because the version I'm
looking at (git master) has nary a reference to the C++ bindings.

Since you say that Gromacs alone compiles fine, I suspect the problem
is that Plumed uses the C++ bindings.  The Plumed download site hosted
by Google Docs (yuck!) is down/broken/in-redirect-hell so I can't
verify this hypothesis right now.

Jeff

On Tue, Jul 9, 2013 at 8:44 AM, Jeff Squyres (jsquyres)
 wrote:
> If you care, the issue is that it looks like Gromacs is using the MPI C++ 
> bindings.  You therefore need to use the MPI C++ wrapper compiler, mpic++ 
> (vs. mpicc, which is the MPI C wrapper compiler).
>
>
> On Jul 9, 2013, at 9:41 AM, Tomek Wlodarski  wrote:
>
>> I used mpicc but when I switched in Makefile to mpic++ it compiled
>> without errors.
>> Thanks a lot!
>> Best,
>>
>> tomek
>>
>> On Tue, Jul 9, 2013 at 2:31 PM, Jeff Squyres (jsquyres)
>>  wrote:
>>> I don't see all the info requested from that web page, but it looks like 
>>> OMPI built the C++ bindings ok.
>>>
>>> Did you use mpic++ to build Gromacs?
>>>
>>>
>>> On Jul 9, 2013, at 9:20 AM, Tomek Wlodarski  
>>> wrote:
>>>
 So I am running OpenMPi1.6.3 (config.log attached)
 And I would like to install gromacs patched with plumed (scientific
 computing). Both uses openmpi.
 Gromacs alone compiles without errors (openMPI works). But when
 patched I got one mentioned before.
 I am sending config file for patched gromacs.
 If you need any other file I would be happy to provide.
 Thanks a lot!
 Best,

 tomek
 ___
 users mailing list
 us...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> --
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to: 
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Jeff Hammond
jeff.scie...@gmail.com

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Tim Carlson]

On Mon, 8 Jul 2013, Tim Carlson wrote:

Now that I have gone through this process, I'll report that it works with 
the caveat that you can't use the openmpi wrappers for compiling. Recall 
that the Phi card does not have either the GNU or Intel compilers 
installed. While you could build up a tool chain for the GNU 
compilers, you're not going to get a native Intel compiler unless Intel 
decides to support it.


Here is the process from end to end to get Openmpi to build a native Phi 
application.


export PATH=/usr/linux-k1om-4.7/bin:$PATH
. /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
export CC="icc -mmic"
export CXX="icpc -mmic"

cd ~
tar zxf openmpi-1.6.4.tar.gz
cd openmpi-1.6.4
./configure --prefix=/people/tim/mic/openmpi/intel \
--build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
--disable-mpi-f77 \
AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib LD=x86_64-k1om-linux-ld
make
make install

That leaves me with a native build of openmpi in 
/people/tim/mic/openmpi/intel


It is of course tempting to just do a
export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
and start using mpicc to build my code but that does not work because:

1) If I try this on the host system I am going to get "wrong architecture" 
because mpicc was build for the Phi and not for the x86_64 host


2) If I try running it on the Phi, I don't have access to "icc" because I 
can't run the compiler directly on the Phi.


I can "cheat" and see what the mpicc command really does by using "mpicc 
--show" for another installation of openmpi and munge the paths correctly. 
In this case


icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
-L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
-lrt -lnsl -lutil -lm -ldl -o cpi.x

That leaves me with a Phi native version of cpi.x which I can then 
execute on the Phi


$ ssh phi002-mic0

( I have NFS mounts on the Phi for all the bits I need )

~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
~ $ export 
LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
~ $ cd mic
~/mic $ mpirun -np 12 cpi.x
Process 7 on phi002-mic0.local
Process 10 on phi002-mic0.local
Process 2 on phi002-mic0.local
Process 9 on phi002-mic0.local
Process 1 on phi002-mic0.local
Process 3 on phi002-mic0.local
Process 11 on phi002-mic0.local
Process 5 on phi002-mic0.local
Process 8 on phi002-mic0.local
Process 4 on phi002-mic0.local
Process 6 on phi002-mic0.local
Process 0 on phi002-mic0.local
pi is approximately 3.1416009869231245, Error is 0.0814
wall clock time = 0.001766



On Mon, 8 Jul 2013, Elken, Tom wrote:

My mistake on the OFED bits. The host I was installing on did not have all of 
the MPSS software installed (my cluster admin node and not one of the compute 
nodes). Adding the intel-mic-ofed-card RPM fixed the problem with compiling 
the btl:openib bits with both the GNU and Intel compilers using the 
cross-compiler route (-mmic on the Intel side)


Still working on getting the resulting mpicc wrapper working on the MIC side. 
When I get a working example I'll post the results.


Thanks!

Tim




  

 Hi Tim,

  

  

 Well, in general and not on MIC I usually build the MPI stacks using the
 Intel compiler set. Have you ran into s/w that requires GCC instead of
 Intel
 compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
 produce
 MIC native code (the OpenMPI stack for that matter)? 

 [Tom]

 Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
 build MIC code.

  

 Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
 failed to produce something with OFED support (not surprising)

 export PATH=/usr/linux-k1om-4.7/bin:$PATH
 ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
 --disable-mpi-f77

 checking if MCA component btl:openib can compile... no

  

 Regarding a Gnu cross compiler, this worked for one of our engineers
 building for True Scale HCAs and PSM/infinipath.  But it may give useful
 tips for building for btl:openib as well:

  

 3. How to configure/compile OpenMPI:

    a). untar the openmpi tarball.

    b). edit configure in top directory, add '-linfinipath' after
 '-lpsm_infinipath "

        but not necessary for messages, only for command lines.

  

    c). run the following script,

 #!/bin/sh

  

 ./configure \

 --host=x86_64-k1om-linux \

 --enable-mpi-f77=no --enable-mpi-f90=no \

 --with-psm=/…/psm-7.6 \

 --prefix=/…/openmpi \

 CC=x86_64-k1om-linux-gcc  CXX=x86_64-k1om-linux-g++ \

 AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib

  

    d). run 'make' and 'make install'

  

 OK, I see that they did not configure for mpi-f77 & mpif90, but perhaps
 this
 is still helpful, if the AR and RANLIB flags are important.

 -Tom

  

  

  

 regards

 Michael

  

 On Mon, Jul 8, 2013 at 4:30 PM,

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Ralph Castain]

Hi Tim

Quick question: can the procs on the MIC communicate with procs on (a) the 
local host, (b) other hosts, and (c) MICs on other hosts?

The last two would depend on having direct access to one or more network 
transports.


On Jul 9, 2013, at 10:18 AM, Tim Carlson  wrote:

> On Mon, 8 Jul 2013, Tim Carlson wrote:
> 
> Now that I have gone through this process, I'll report that it works with the 
> caveat that you can't use the openmpi wrappers for compiling. Recall that the 
> Phi card does not have either the GNU or Intel compilers installed. While you 
> could build up a tool chain for the GNU compilers, you're not going to get a 
> native Intel compiler unless Intel decides to support it.
> 
> Here is the process from end to end to get Openmpi to build a native Phi 
> application.
> 
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
> export CC="icc -mmic"
> export CXX="icpc -mmic"
> 
> cd ~
> tar zxf openmpi-1.6.4.tar.gz
> cd openmpi-1.6.4
> ./configure --prefix=/people/tim/mic/openmpi/intel \
> --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77 \
> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib 
> LD=x86_64-k1om-linux-ld
> make
> make install
> 
> That leaves me with a native build of openmpi in /people/tim/mic/openmpi/intel
> 
> It is of course tempting to just do a
> export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
> and start using mpicc to build my code but that does not work because:
> 
> 1) If I try this on the host system I am going to get "wrong architecture" 
> because mpicc was build for the Phi and not for the x86_64 host
> 
> 2) If I try running it on the Phi, I don't have access to "icc" because I 
> can't run the compiler directly on the Phi.
> 
> I can "cheat" and see what the mpicc command really does by using "mpicc 
> --show" for another installation of openmpi and munge the paths correctly. In 
> this case
> 
> icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
> -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
> -lrt -lnsl -lutil -lm -ldl -o cpi.x
> 
> That leaves me with a Phi native version of cpi.x which I can then execute on 
> the Phi
> 
> $ ssh phi002-mic0
> 
> ( I have NFS mounts on the Phi for all the bits I need )
> 
> ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
> ~ $ export 
> LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
> ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
> ~ $ cd mic
> ~/mic $ mpirun -np 12 cpi.x
> Process 7 on phi002-mic0.local
> Process 10 on phi002-mic0.local
> Process 2 on phi002-mic0.local
> Process 9 on phi002-mic0.local
> Process 1 on phi002-mic0.local
> Process 3 on phi002-mic0.local
> Process 11 on phi002-mic0.local
> Process 5 on phi002-mic0.local
> Process 8 on phi002-mic0.local
> Process 4 on phi002-mic0.local
> Process 6 on phi002-mic0.local
> Process 0 on phi002-mic0.local
> pi is approximately 3.1416009869231245, Error is 0.0814
> wall clock time = 0.001766
> 
> 
>> On Mon, 8 Jul 2013, Elken, Tom wrote:
>> 
>> My mistake on the OFED bits. The host I was installing on did not have all 
>> of the MPSS software installed (my cluster admin node and not one of the 
>> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
>> compiling the btl:openib bits with both the GNU and Intel compilers using 
>> the cross-compiler route (-mmic on the Intel side)
>> 
>> Still working on getting the resulting mpicc wrapper working on the MIC 
>> side. When I get a working example I'll post the results.
>> 
>> Thanks!
>> 
>> Tim
>> 
>> 
>>> 
>>>  
>>> 
>>> Hi Tim,
>>> 
>>>  
>>> 
>>>  
>>> 
>>> Well, in general and not on MIC I usually build the MPI stacks using the
>>> Intel compiler set. Have you ran into s/w that requires GCC instead of
>>> Intel
>>> compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
>>> produce
>>> MIC native code (the OpenMPI stack for that matter)? 
>>> 
>>> [Tom]
>>> 
>>> Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
>>> build MIC code.
>>> 
>>>  
>>> 
>>> Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
>>> failed to produce something with OFED support (not surprising)
>>> 
>>> export PATH=/usr/linux-k1om-4.7/bin:$PATH
>>> ./configure --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
>>> --disable-mpi-f77
>>> 
>>> checking if MCA component btl:openib can compile... no
>>> 
>>>  
>>> 
>>> Regarding a Gnu cross compiler, this worked for one of our engineers
>>> building for True Scale HCAs and PSM/infinipath.  But it may give useful
>>> tips for building for btl:openib as well:
>>> 
>>>  
>>> 
>>> 3. How to configure/compile OpenMPI:
>>> 
>>>a). untar the openmpi tarball.
>>> 
>>>b). edit configure in top directory, add '-linfinipath' after
>>> '-lpsm_infinipath "
>>> 
>>>but no

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Daniels, Marcus G]

The Intel MPI implementation does this.  The performance between the 
accelerators and the host is poor though.   About 20mb/sec in my ping/pong 
test.   Intra-MIC communication is about a 1GB/sec whereas intra-host is about 
6GB/sec.  Latency is higher (i.e. worse) for the intra-MIC communication too 
(vs. intra-host) by about the same factor.  

Thanks Tim for the hints on building OpenMPI.  

From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] on behalf of 
Ralph Castain [r...@open-mpi.org]
Sent: Tuesday, July 09, 2013 12:14 PM
To: Tim Carlson; Open MPI Users
Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5
an 1.7.2

Hi Tim

Quick question: can the procs on the MIC communicate with procs on (a) the 
local host, (b) other hosts, and (c) MICs on other hosts?

The last two would depend on having direct access to one or more network 
transports.


On Jul 9, 2013, at 10:18 AM, Tim Carlson  wrote:

> On Mon, 8 Jul 2013, Tim Carlson wrote:
>
> Now that I have gone through this process, I'll report that it works with the 
> caveat that you can't use the openmpi wrappers for compiling. Recall that the 
> Phi card does not have either the GNU or Intel compilers installed. While you 
> could build up a tool chain for the GNU compilers, you're not going to get a 
> native Intel compiler unless Intel decides to support it.
>
> Here is the process from end to end to get Openmpi to build a native Phi 
> application.
>
> export PATH=/usr/linux-k1om-4.7/bin:$PATH
> . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
> export CC="icc -mmic"
> export CXX="icpc -mmic"
>
> cd ~
> tar zxf openmpi-1.6.4.tar.gz
> cd openmpi-1.6.4
> ./configure --prefix=/people/tim/mic/openmpi/intel \
> --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77 \
> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib 
> LD=x86_64-k1om-linux-ld
> make
> make install
>
> That leaves me with a native build of openmpi in /people/tim/mic/openmpi/intel
>
> It is of course tempting to just do a
> export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
> and start using mpicc to build my code but that does not work because:
>
> 1) If I try this on the host system I am going to get "wrong architecture" 
> because mpicc was build for the Phi and not for the x86_64 host
>
> 2) If I try running it on the Phi, I don't have access to "icc" because I 
> can't run the compiler directly on the Phi.
>
> I can "cheat" and see what the mpicc command really does by using "mpicc 
> --show" for another installation of openmpi and munge the paths correctly. In 
> this case
>
> icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
> -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
> -lrt -lnsl -lutil -lm -ldl -o cpi.x
>
> That leaves me with a Phi native version of cpi.x which I can then execute on 
> the Phi
>
> $ ssh phi002-mic0
>
> ( I have NFS mounts on the Phi for all the bits I need )
>
> ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
> ~ $ export 
> LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
> ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
> ~ $ cd mic
> ~/mic $ mpirun -np 12 cpi.x
> Process 7 on phi002-mic0.local
> Process 10 on phi002-mic0.local
> Process 2 on phi002-mic0.local
> Process 9 on phi002-mic0.local
> Process 1 on phi002-mic0.local
> Process 3 on phi002-mic0.local
> Process 11 on phi002-mic0.local
> Process 5 on phi002-mic0.local
> Process 8 on phi002-mic0.local
> Process 4 on phi002-mic0.local
> Process 6 on phi002-mic0.local
> Process 0 on phi002-mic0.local
> pi is approximately 3.1416009869231245, Error is 0.0814
> wall clock time = 0.001766
>
>
>> On Mon, 8 Jul 2013, Elken, Tom wrote:
>>
>> My mistake on the OFED bits. The host I was installing on did not have all 
>> of the MPSS software installed (my cluster admin node and not one of the 
>> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
>> compiling the btl:openib bits with both the GNU and Intel compilers using 
>> the cross-compiler route (-mmic on the Intel side)
>>
>> Still working on getting the resulting mpicc wrapper working on the MIC 
>> side. When I get a working example I'll post the results.
>>
>> Thanks!
>>
>> Tim
>>
>>
>>>
>>>
>>>
>>> Hi Tim,
>>>
>>>
>>>
>>>
>>>
>>> Well, in general and not on MIC I usually build the MPI stacks using the
>>> Intel compiler set. Have you ran into s/w that requires GCC instead of
>>> Intel
>>> compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
>>> produce
>>> MIC native code (the OpenMPI stack for that matter)?
>>>
>>> [Tom]
>>>
>>> Good idea Michael,  With the Intel Compiler, I would use the -mmic flag to
>>> build MIC code.
>>>
>>>
>>>
>>> Tim wrote: “My first pass at doing a cross-compile with the GNU compilers
>>> failed to produce something with OFED support (not

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Ralph Castain]

Understood - but I was wondering if that was true for OMPI as well.

On Jul 9, 2013, at 11:30 AM, "Daniels, Marcus G"  wrote:

> The Intel MPI implementation does this.  The performance between the 
> accelerators and the host is poor though.   About 20mb/sec in my ping/pong 
> test.   Intra-MIC communication is about a 1GB/sec whereas intra-host is 
> about 6GB/sec.  Latency is higher (i.e. worse) for the intra-MIC 
> communication too (vs. intra-host) by about the same factor.  
> 
> Thanks Tim for the hints on building OpenMPI.  
> 
> From: users-boun...@open-mpi.org [users-boun...@open-mpi.org] on behalf of 
> Ralph Castain [r...@open-mpi.org]
> Sent: Tuesday, July 09, 2013 12:14 PM
> To: Tim Carlson; Open MPI Users
> Subject: Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5  
>   an 1.7.2
> 
> Hi Tim
> 
> Quick question: can the procs on the MIC communicate with procs on (a) the 
> local host, (b) other hosts, and (c) MICs on other hosts?
> 
> The last two would depend on having direct access to one or more network 
> transports.
> 
> 
> On Jul 9, 2013, at 10:18 AM, Tim Carlson  wrote:
> 
>> On Mon, 8 Jul 2013, Tim Carlson wrote:
>> 
>> Now that I have gone through this process, I'll report that it works with 
>> the caveat that you can't use the openmpi wrappers for compiling. Recall 
>> that the Phi card does not have either the GNU or Intel compilers installed. 
>> While you could build up a tool chain for the GNU compilers, you're not 
>> going to get a native Intel compiler unless Intel decides to support it.
>> 
>> Here is the process from end to end to get Openmpi to build a native Phi 
>> application.
>> 
>> export PATH=/usr/linux-k1om-4.7/bin:$PATH
>> . /share/apps/intel/composer_xe_2013.3.163/bin/iccvars.sh intel64
>> export CC="icc -mmic"
>> export CXX="icpc -mmic"
>> 
>> cd ~
>> tar zxf openmpi-1.6.4.tar.gz
>> cd openmpi-1.6.4
>> ./configure --prefix=/people/tim/mic/openmpi/intel \
>> --build=x86_64-unknown-linux-gnu --host=x86_64-k1om-linux \
>> --disable-mpi-f77 \
>> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-ranlib 
>> LD=x86_64-k1om-linux-ld
>> make
>> make install
>> 
>> That leaves me with a native build of openmpi in 
>> /people/tim/mic/openmpi/intel
>> 
>> It is of course tempting to just do a
>> export PATH=/people/tim/mic/openmpi/intel/bin:$PATH
>> and start using mpicc to build my code but that does not work because:
>> 
>> 1) If I try this on the host system I am going to get "wrong architecture" 
>> because mpicc was build for the Phi and not for the x86_64 host
>> 
>> 2) If I try running it on the Phi, I don't have access to "icc" because I 
>> can't run the compiler directly on the Phi.
>> 
>> I can "cheat" and see what the mpicc command really does by using "mpicc 
>> --show" for another installation of openmpi and munge the paths correctly. 
>> In this case
>> 
>> icc -mmic cpi.c -I/people/tim/mic/openmpi/intel/include -pthread \
>> -L/people/tim/mic/openmpi/intel/lib -lmpi -ldl -lm -Wl,--export-dynamic \
>> -lrt -lnsl -lutil -lm -ldl -o cpi.x
>> 
>> That leaves me with a Phi native version of cpi.x which I can then execute 
>> on the Phi
>> 
>> $ ssh phi002-mic0
>> 
>> ( I have NFS mounts on the Phi for all the bits I need )
>> 
>> ~ $ export PATH=/people/tim/mic/openmpi/intel/bin/:$PATH
>> ~ $ export 
>> LD_LIBRARY_PATH=/share/apps/intel/composer_xe_2013.3.163/compiler/lib/mic/
>> ~ $ export LD_LIBRARY_PATH=/people/tim/mic/openmpi/intel/lib:$LD_LIBRARY_PATH
>> ~ $ cd mic
>> ~/mic $ mpirun -np 12 cpi.x
>> Process 7 on phi002-mic0.local
>> Process 10 on phi002-mic0.local
>> Process 2 on phi002-mic0.local
>> Process 9 on phi002-mic0.local
>> Process 1 on phi002-mic0.local
>> Process 3 on phi002-mic0.local
>> Process 11 on phi002-mic0.local
>> Process 5 on phi002-mic0.local
>> Process 8 on phi002-mic0.local
>> Process 4 on phi002-mic0.local
>> Process 6 on phi002-mic0.local
>> Process 0 on phi002-mic0.local
>> pi is approximately 3.1416009869231245, Error is 0.0814
>> wall clock time = 0.001766
>> 
>> 
>>> On Mon, 8 Jul 2013, Elken, Tom wrote:
>>> 
>>> My mistake on the OFED bits. The host I was installing on did not have all 
>>> of the MPSS software installed (my cluster admin node and not one of the 
>>> compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem with 
>>> compiling the btl:openib bits with both the GNU and Intel compilers using 
>>> the cross-compiler route (-mmic on the Intel side)
>>> 
>>> Still working on getting the resulting mpicc wrapper working on the MIC 
>>> side. When I get a working example I'll post the results.
>>> 
>>> Thanks!
>>> 
>>> Tim
>>> 
>>> 
 
 
 
 Hi Tim,
 
 
 
 
 
 Well, in general and not on MIC I usually build the MPI stacks using the
 Intel compiler set. Have you ran into s/w that requires GCC instead of
 Intel
 compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
 produce

Re: [OMPI users] Support for CUDA and GPU-direct with OpenMPI 1.6.5 an 1.7.2

[Michael Thomadakis]

Tim, thanks for trying this out ...

Now you should be able to let part of the same OpenMPI application run on
the host multi-core side and the other part on the MIC. IntelMPI can do
this using an MPMD command line where the Xeon binaries run on the host,
whereas the MIC ones on MIC card(s).

I guess you should be able to directly do this from the same OpenMPI mpirun
command line ...

thanks
Michael


On Tue, Jul 9, 2013 at 12:18 PM, Tim Carlson  wrote:

> On Mon, 8 Jul 2013, Tim Carlson wrote:
>
> Now that I have gone through this process, I'll report that it works with
> the caveat that you can't use the openmpi wrappers for compiling. Recall
> that the Phi card does not have either the GNU or Intel compilers
> installed. While you could build up a tool chain for the GNU compilers,
> you're not going to get a native Intel compiler unless Intel decides to
> support it.
>
> Here is the process from end to end to get Openmpi to build a native Phi
> application.
>
> export PATH=/usr/linux-k1om-4.7/bin:$**PATH
> . /share/apps/intel/composer_xe_**2013.3.163/bin/iccvars.sh intel64
> export CC="icc -mmic"
> export CXX="icpc -mmic"
>
> cd ~
> tar zxf openmpi-1.6.4.tar.gz
> cd openmpi-1.6.4
> ./configure --prefix=/people/tim/mic/**openmpi/intel \
> --build=x86_64-unknown-linux-**gnu --host=x86_64-k1om-linux \
> --disable-mpi-f77 \
> AR=x86_64-k1om-linux-ar RANLIB=x86_64-k1om-linux-**ranlib
> LD=x86_64-k1om-linux-ld
> make
> make install
>
> That leaves me with a native build of openmpi in
> /people/tim/mic/openmpi/intel
>
> It is of course tempting to just do a
> export PATH=/people/tim/mic/openmpi/**intel/bin:$PATH
> and start using mpicc to build my code but that does not work because:
>
> 1) If I try this on the host system I am going to get "wrong architecture"
> because mpicc was build for the Phi and not for the x86_64 host
>
> 2) If I try running it on the Phi, I don't have access to "icc" because I
> can't run the compiler directly on the Phi.
>
> I can "cheat" and see what the mpicc command really does by using "mpicc
> --show" for another installation of openmpi and munge the paths correctly.
> In this case
>
> icc -mmic cpi.c -I/people/tim/mic/openmpi/**intel/include -pthread \
> -L/people/tim/mic/openmpi/**intel/lib -lmpi -ldl -lm -Wl,--export-dynamic
> \
> -lrt -lnsl -lutil -lm -ldl -o cpi.x
>
> That leaves me with a Phi native version of cpi.x which I can then execute
> on the Phi
>
> $ ssh phi002-mic0
>
> ( I have NFS mounts on the Phi for all the bits I need )
>
> ~ $ export PATH=/people/tim/mic/openmpi/**intel/bin/:$PATH
> ~ $ export LD_LIBRARY_PATH=/share/apps/**intel/composer_xe_2013.3.163/**
> compiler/lib/mic/
> ~ $ export LD_LIBRARY_PATH=/people/tim/**mic/openmpi/intel/lib:$LD_**
> LIBRARY_PATH
> ~ $ cd mic
> ~/mic $ mpirun -np 12 cpi.x
> Process 7 on phi002-mic0.local
> Process 10 on phi002-mic0.local
> Process 2 on phi002-mic0.local
> Process 9 on phi002-mic0.local
> Process 1 on phi002-mic0.local
> Process 3 on phi002-mic0.local
> Process 11 on phi002-mic0.local
> Process 5 on phi002-mic0.local
> Process 8 on phi002-mic0.local
> Process 4 on phi002-mic0.local
> Process 6 on phi002-mic0.local
> Process 0 on phi002-mic0.local
> pi is approximately 3.1416009869231245, Error is 0.0814
> wall clock time = 0.001766
>
>
>
>  On Mon, 8 Jul 2013, Elken, Tom wrote:
>>
>> My mistake on the OFED bits. The host I was installing on did not have
>> all of the MPSS software installed (my cluster admin node and not one of
>> the compute nodes). Adding the intel-mic-ofed-card RPM fixed the problem
>> with compiling the btl:openib bits with both the GNU and Intel compilers
>> using the cross-compiler route (-mmic on the Intel side)
>>
>> Still working on getting the resulting mpicc wrapper working on the MIC
>> side. When I get a working example I'll post the results.
>>
>> Thanks!
>>
>> Tim
>>
>>
>>
>>>
>>>
>>>  Hi Tim,
>>>
>>>
>>>
>>>
>>>
>>>  Well, in general and not on MIC I usually build the MPI stacks using the
>>>  Intel compiler set. Have you ran into s/w that requires GCC instead of
>>>  Intel
>>>  compilers (beside Nvidia Cuda)? Did you try to use Intel compiler to
>>>  produce
>>>  MIC native code (the OpenMPI stack for that matter)?
>>>
>>>  [Tom]
>>>
>>>  Good idea Michael,  With the Intel Compiler, I would use the -mmic flag
>>> to
>>>  build MIC code.
>>>
>>>
>>>
>>>  Tim wrote: “My first pass at doing a cross-compile with the GNU
>>> compilers
>>>  failed to produce something with OFED support (not surprising)
>>>
>>>  export PATH=/usr/linux-k1om-4.7/bin:$**PATH
>>>  ./configure --build=x86_64-unknown-linux-**gnu
>>> --host=x86_64-k1om-linux \
>>>  --disable-mpi-f77
>>>
>>>  checking if MCA component btl:openib can compile... no
>>>
>>>
>>>
>>>  Regarding a Gnu cross compiler, this worked for one of our engineers
>>>  building for True Scale HCAs and PSM/infinipath.  But it may give useful
>>>  tips for building for btl:openib as well:
>>>
>>>
>>>
>>>  3. How to configure/compile 

Re: [OMPI users] using the xrc queues

[Mike Dubman]

Hi,
I would suggest use MXM (part of mofed, can be downloaded as standalone rpm
from http://mellanox.com/products/mxm for ofed)

It uses UD (constant memory footprint) and should provide good performance.
The next MXM v2.0 will support RC and DC (reliable UD) as well.

Once mxm is installed from rpm (or extracted elsewhere from rpm->tarball) -
you can point ompi configure with "--with-mxm=/path/to/mxm")
Regards
M


On Fri, Jul 5, 2013 at 10:33 PM, Ben  wrote:

> I'm part of a team that maintains a global climate model running under
> mpi. Recently we have been trying it out with different mpi stacks
> at high resolution/processor counts.
> At one point in the code there is a large number of mpi_isends/mpi_recv
> (tens to hundreds of thousands) when data distributed across all mpi
> processes must be collective on a particular processor or processors be
> transformed to a new resolution before writing. At first the model was
> crashing with a message:
> "A process failed to create a queue pair. This usually means either the
> device has run out of queue pairs (too many connections) or there are
> insufficient resources available to allocate a queue pair (out of memory).
> The latter can happen if either 1) insufficient memory is available, or 2)
> no more physical memory can be registered with the device."
> when it hit the part of code with the send/receives. Watching the node
> memory in an xterm I could see the memory skyrocket and fill the node.
>
> Somewhere we found a suggestion try using the xrc queues (
> http://www.open-mpi.org/faq/?**category=openfabrics#ib-xrc)
> to get around this problem and indeed running with
>
> setenv OMPI_MCA_btl_openib_receive_**queues "X,128,256,192,128:X,2048,256,
> **128,32:X,12288,256,128,32:X,**65536,256,128,32"
> mpirun --bind-to-core -np numproc ./app
>
> allowed the model to successfully run. It still seems to use a large
> amount of memory when it writes (on the order of several Gb). Does anyone
> have any  suggestions on how to perhaps tweak the settings to help with
> memory use.
>
> --
> Ben Auer, PhD   SSAI, Scientific Programmer/Analyst
> NASA GSFC,  Global Modeling and Assimilation Office
> Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD  20771
> Phone: 301-286-9176   Fax: 301-614-6246
>
> __**_
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/**mailman/listinfo.cgi/users
>