Re: [OMPI users] OpenMPI 1.6.1 with Intel Cluster Studio 2012

2012-09-28 Thread Paul Edmon
Resolution to this. Upgrading to OpenMPI 1.6.2 and getting Intel Cluster Studio 2013 did the trick. -Paul Edmon- On 9/8/2012 4:59 PM, Paul Edmon wrote: Interesting. I figured that might be the case. I will have to contact Intel and find out if we can get a newer version. Thanks. -Paul Edm

[OMPI users] Mellanox MLX4_EVENT_TYPE_SRQ_LIMIT kernel messages

2012-09-28 Thread Mark Dixon
Hi, We've been putting a new Mellanox QDR Intel Sandy Bridge cluster, based on CentOS 6.3, through its paces and we're getting repeated kernel messages we never used to get on CentOS 5. An example on one node: Sep 28 09:58:20 g8s1n2 kernel: mlx4_core :01:00.0: mlx4_eq_int: MLX4_EVENT_TYP

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Sébastien Boisvert
On 28/09/12 10:50 AM, Jeff Squyres wrote: > On Sep 28, 2012, at 10:38 AM, Sébastien Boisvert wrote: > >> 1.5 us is very good. But I get 1.5 ms with shared queues (see above). > > Oh, I mis-read (I blame it on jet-lag...). > > Yes, that seems wy too high. > > You didn't do a developer build,

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Sébastien Boisvert
I totally agree. Portability is the most important thing for me. That's why the virtual router is not required to run the software. On 28/09/12 10:32 AM, Iliev, Hristo wrote: >>From my understanding of the Open MPI code, it uses negative tags to >>implement collective operations and one-sided co

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Jeff Squyres
On Sep 28, 2012, at 10:38 AM, Sébastien Boisvert wrote: > 1.5 us is very good. But I get 1.5 ms with shared queues (see above). Oh, I mis-read (I blame it on jet-lag...). Yes, that seems wy too high. You didn't do a developer build, did you? We add a bunch of extra debugging in developer

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Sébastien Boisvert
Hello, On 28/09/12 10:00 AM, Jeff Squyres wrote: > On Sep 28, 2012, at 9:50 AM, Sébastien Boisvert wrote: > >> I did not know about shared queues. >> >> It does not run out of memory. ;-) > > It runs out of *registered* memory, which could be far less than your actual > RAM. Check this FAQ ite

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Iliev, Hristo
>From my understanding of the Open MPI code, it uses negative tags to implement >collective operations and one-sided communication only. Your code might work >if you use neither of them but will not be portable, not only across MPI >implementations, but also some future change in Open MPI might

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Jeff Squyres
On Sep 28, 2012, at 9:50 AM, Sébastien Boisvert wrote: > I did not know about shared queues. > > It does not run out of memory. ;-) It runs out of *registered* memory, which could be far less than your actual RAM. Check this FAQ item in particular: http://www.open-mpi.org/faq/?category=op

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Sébastien Boisvert
Hi, I did not know about shared queues. It does not run out of memory. ;-) But the latency is not very good. ** Test 1 --mca btl_openib_max_send_size 4096 \ --mca btl_openib_eager_limit 4096 \ --mca btl_openib_rndv_eager_limit 4096 \ --mca btl_openib_receive_queues S,4096,2048,1024,32 \ I ge

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Jeff Squyres
Beware that Open MPI uses negative tags for internal uses. You might conflict with that sometimes. On Sep 28, 2012, at 9:08 AM, Sébastien Boisvert wrote: > Hello, > > My application has 191 MPI tags allocated with allocateMessageTagHandle, so > 7 bits is not enough. > > Indeed, tags can be va

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Sébastien Boisvert
Hello, My application has 191 MPI tags allocated with allocateMessageTagHandle, so 7 bits is not enough. Indeed, tags can be valued from 0 to 2147483647 inclusively with this MPI_TAG_UB value in Open-MPI. I misused the returned pointer. In Open-MPI, MPI_ANY_TAG is -1. I removed the boundary chec

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Jeff Squyres
On Sep 27, 2012, at 7:22 PM, Sébastien Boisvert wrote: > Without the virtual message router, I get messages like these: > > [cp2558][[30209,1],0][connect/btl_openib_connect_oob.c:490:qp_create_one] > error creating qp errno says Cannot allocate memory You're running out of registered memory. C

Re: [OMPI users] static, standalone mpirun (Jeff Squyres)

2012-09-28 Thread Ilias Miroslav
Dear Jeff, here I meant "" simply my local installation space... > ./configure --prefix= --without-memory-manager CXX=icpc CC=icc > F77=ifort FC=ifort LDFLAGS=--static --disable-shared --enable-static In between I managed to fix the relocated static, standalone mpirun (v 1.6.1): beyond the mpir

Re: [OMPI users] About MPI_TAG_UB

2012-09-28 Thread Iliev, Hristo
Hello, MPI_TAG_UB in Open MPI is INT_MAX == 2^31-1 == 2147483647. The value of 17438272 (0x10A1640) is a bit strange for MPI_TAG_UB. I would rather say that it is the value of a pointer to someplace in the heap, i.e. you have missed the fact that the attribute value as returned by MPI_Comm_get_att