[OMPI users] progressing passive-sync RMA operations

[Christopher O'Grady]

Hi,

I'm using openmpi 1.8.1 and could use some brief advice on RMA with
passive synchronization.  I've been trying to use that synchronization
because I can't guarantee that the two ranks that are communicating
have matching pairs of Start/Complete Post/Wait required by active
synchronization.  On the "target" side, I think I've learned that I
have to enter the MPI library in order to "progress" the RMA request
as described here:

http://stackoverflow.com/questions/18737545/mpi-with-c-passive-rma-synchronization

So to "enter the MPI library" I've tried calling MPI_Iprobe on the
target side intermittently to try to progress the RMA, but I don't
believe that works.  Is there another routine that would work better?
I guess I could try "enable-opal-multi-threads" recommended in the
above post, but was worried about thread-safety, and thought I would
first ask for advice here on the cleanest way to progress RMA requests
on the target with passive synchronization.

Thanks for any quick thoughts you might have!

chris

Re: [OMPI users] send and receive vectors + variable length

[Diego Avesani]

Dear George, Dear Jeff, Dear All,

Thanks Thanks a lot

Here, the new version of the program. Now there is only one barrier. There
is no more allocate\deallocate in the receive part.

What do you think? Is all right? did I miss something or I need to improve
something else?

I have not complete understood your point Jeff, about WAITALL, eve looking
at the examples.

George what do you mean?

Again Thanks a lot


Diego


On 9 January 2015 at 00:58, George Bosilca  wrote:

> I'm confused by this statement. The examples pointed to are handling
> blocking sends and receives, while this example is purely based on
> non-blocking communications. In this particular case I see no hard of
> waiting on the requests in any random order as long as all of them are
> posted before the first wait.
>
>   George.
>
>
> On Thu, Jan 8, 2015 at 5:24 PM, Jeff Squyres (jsquyres) <
> jsquy...@cisco.com> wrote:
>
>> Also, you are calling WAITALL on all your sends and then WAITALL on all
>> your receives.  This is also incorrect and may deadlock.
>>
>> WAITALL on *all* your pending requests (sends and receives -- put them
>> all in a single array).
>>
>> Look at examples 3.8 and 3.9 in the MPI-3.0 document.
>>
>>
>>
>> On Jan 8, 2015, at 5:15 PM, George Bosilca  wrote:
>>
>> > Diego,
>> >
>> > Non-blocking communications only indicate a communication will happen,
>> it does not force them to happen. They will only complete on the
>> corresponding MPI_Wait, which also marks the moment starting from where the
>> data can be safely altered or accessed (in the case of the MPI_Irecv). Thus
>> deallocating your buffer after the MPI_Isend and MPI_Irecv is incorrect.
>> Also printing the supposedly received values (line 127) is incorrect as
>> there is no reason to have the non-blocking receive completed at that
>> moment.
>> >
>> >   George.
>> >
>> >
>> > On Thu, Jan 8, 2015 at 5:06 PM, Diego Avesani 
>> wrote:
>> > Dear Tom, Dear Jeff, Dear all,
>> > Thanks again
>> >
>> > for Tom:
>> > you are right, I fixed it.
>> >
>> > for Jeff:
>> > if I do not insert the CALL MPI_BARRIER(MPI_COMM_WORLD, MPIdata%iErr)
>> > in the line 112, the program does not stop.
>> >
>> > Am I right?
>> > Here the new version
>> >
>> >
>> >
>> > Diego
>> >
>> >
>> > On 8 January 2015 at 21:12, Tom Rosmond  wrote:
>> > With array bounds checking your program returns an out-of-bounds error
>> > in the mpi_isend call at line 104.  Looks like 'send_request' should be
>> > indexed with 'sendcount', not 'icount'.
>> >
>> > T. Rosmond
>> >
>> >
>> >
>> > On Thu, 2015-01-08 at 20:28 +0100, Diego Avesani wrote:
>> > > the attachment
>> > >
>> > > Diego
>> > >
>> > >
>> > >
>> > > On 8 January 2015 at 19:44, Diego Avesani 
>> > > wrote:
>> > > Dear all,
>> > > I found the error.
>> > > There is a  Ndata2send(iCPU) instead of Ndata2recv(iCPU).
>> > > In the attachment there is the correct version of the program.
>> > >
>> > >
>> > > Only one thing, could do you check if the use of MPI_WAITALL
>> > > and MPI_BARRIER is correct?
>> > >
>> > >
>> > > Thanks again
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Diego
>> > >
>> > >
>> > >
>> > > On 8 January 2015 at 18:48, Diego Avesani
>> > >  wrote:
>> > > Dear all,
>> > > thanks thank a lot, I am learning a lot.
>> > >
>> > >
>> > >
>> > > I have written a simple program that send vectors of
>> > > integers from a CPU to another.
>> > >
>> > >
>> > > The program is written (at least for now) for 4 CPU.
>> > >
>> > >
>> > > The program is quite simple:
>> > > Each CPU knows how many data has to send to the other
>> > > CPUs. This info is than send to the other CPUS. In
>> > > this way each CPU knows how may data has to receive
>> > > from other CPUs.
>> > >
>> > >
>> > > This part of the program works.
>> > >
>> > >
>> > > The problem is in the second part.
>> > >
>> > >
>> > > In the second part, each processor sends a vector of
>> > > integer to the other processor. The size is given and
>> > > each CPU knows the size of the incoming vector form
>> > > the first part of the program.
>> > >
>> > >
>> > > In this second part the program fails and I do not
>> > > know why.
>> > >
>> > >
>> > > In the attachment you can find the program. Could you
>> > > please help me. Problably I didn't understand properly
>> > > the ISEND and IRECV subroutine.
>> > >
>> > >
>> > > Thanks again
>> > >
>> > >
>> > >
>> > > Diego
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > ___
>> > > users mailing list
>> > > us...@open-mpi.org
>> > > Subscription: http://www.op

Re: [OMPI users] send and receive vectors + variable length

[Jeff Squyres (jsquyres)]

Yes, I know examples 3.8/3.9 are blocking examples.

But it's morally the same as:

MPI_WAITALL(send_requests...)
MPI_WAITALL(recv_requests...)

Strictly speaking, that can deadlock, too.  

It reality, it has far less chance of deadlocking than examples 3.8 and 3.9 
(because you're likely within the general progression engine, and the 
implementation will progress both the send and receive requests while in the 
first WAITALL).  

But still, it would be valid for an implementation to *only* progress the send 
requests -- and NOT the receive requests -- while in the first WAITALL.  Which 
makes it functionally equivalent to examples 3.8/3.9.



On Jan 8, 2015, at 6:58 PM, George Bosilca  wrote:

> I'm confused by this statement. The examples pointed to are handling blocking 
> sends and receives, while this example is purely based on non-blocking 
> communications. In this particular case I see no hard of waiting on the 
> requests in any random order as long as all of them are posted before the 
> first wait.
> 
>   George.
> 
> 
> On Thu, Jan 8, 2015 at 5:24 PM, Jeff Squyres (jsquyres)  
> wrote:
> Also, you are calling WAITALL on all your sends and then WAITALL on all your 
> receives.  This is also incorrect and may deadlock.
> 
> WAITALL on *all* your pending requests (sends and receives -- put them all in 
> a single array).
> 
> Look at examples 3.8 and 3.9 in the MPI-3.0 document.
> 
> 
> 
> On Jan 8, 2015, at 5:15 PM, George Bosilca  wrote:
> 
> > Diego,
> >
> > Non-blocking communications only indicate a communication will happen, it 
> > does not force them to happen. They will only complete on the corresponding 
> > MPI_Wait, which also marks the moment starting from where the data can be 
> > safely altered or accessed (in the case of the MPI_Irecv). Thus 
> > deallocating your buffer after the MPI_Isend and MPI_Irecv is incorrect. 
> > Also printing the supposedly received values (line 127) is incorrect as 
> > there is no reason to have the non-blocking receive completed at that 
> > moment.
> >
> >   George.
> >
> >
> > On Thu, Jan 8, 2015 at 5:06 PM, Diego Avesani  
> > wrote:
> > Dear Tom, Dear Jeff, Dear all,
> > Thanks again
> >
> > for Tom:
> > you are right, I fixed it.
> >
> > for Jeff:
> > if I do not insert the CALL MPI_BARRIER(MPI_COMM_WORLD, MPIdata%iErr)
> > in the line 112, the program does not stop.
> >
> > Am I right?
> > Here the new version
> >
> >
> >
> > Diego
> >
> >
> > On 8 January 2015 at 21:12, Tom Rosmond  wrote:
> > With array bounds checking your program returns an out-of-bounds error
> > in the mpi_isend call at line 104.  Looks like 'send_request' should be
> > indexed with 'sendcount', not 'icount'.
> >
> > T. Rosmond
> >
> >
> >
> > On Thu, 2015-01-08 at 20:28 +0100, Diego Avesani wrote:
> > > the attachment
> > >
> > > Diego
> > >
> > >
> > >
> > > On 8 January 2015 at 19:44, Diego Avesani 
> > > wrote:
> > > Dear all,
> > > I found the error.
> > > There is a  Ndata2send(iCPU) instead of Ndata2recv(iCPU).
> > > In the attachment there is the correct version of the program.
> > >
> > >
> > > Only one thing, could do you check if the use of MPI_WAITALL
> > > and MPI_BARRIER is correct?
> > >
> > >
> > > Thanks again
> > >
> > >
> > >
> > >
> > >
> > > Diego
> > >
> > >
> > >
> > > On 8 January 2015 at 18:48, Diego Avesani
> > >  wrote:
> > > Dear all,
> > > thanks thank a lot, I am learning a lot.
> > >
> > >
> > >
> > > I have written a simple program that send vectors of
> > > integers from a CPU to another.
> > >
> > >
> > > The program is written (at least for now) for 4 CPU.
> > >
> > >
> > > The program is quite simple:
> > > Each CPU knows how many data has to send to the other
> > > CPUs. This info is than send to the other CPUS. In
> > > this way each CPU knows how may data has to receive
> > > from other CPUs.
> > >
> > >
> > > This part of the program works.
> > >
> > >
> > > The problem is in the second part.
> > >
> > >
> > > In the second part, each processor sends a vector of
> > > integer to the other processor. The size is given and
> > > each CPU knows the size of the incoming vector form
> > > the first part of the program.
> > >
> > >
> > > In this second part the program fails and I do not
> > > know why.
> > >
> > >
> > > In the attachment you can find the program. Could you
> > > please help me. Problably I didn't understand properly
> > > the ISEND and IRECV subroutine.
> > >
> > >
> > > Thanks again
> > >
> > >
> > >
> > > Diego
> > >
> > >
> > >
> > >
> > >
> > >
> > > __

Re: [OMPI users] libpsm_infinipath issues?

[VanEss.Michael]

Hi Gus, Andrew, list

No, I haven't tried 1.8.4.  I'll give it a whirl and hope that it fixes the 
issue.

Thanks,

Mike

> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Gus Correa
> Sent: Thursday, January 08, 2015 4:45 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] libpsm_infinipath issues?
>
> Hi Michael, Andrew, list
>
> knem is doesn't work in OMPI 1.8.3.
> See this thread:
> http://www.open-mpi.org/community/lists/users/2014/10/25511.php
>
> A fix was promised on OMPI 1.8.4:
> http://www.open-mpi.org/software/ompi/v1.8/
>
> Have you tried it?
>
> I hope this helps,
> Gus Correa
>
> On 01/08/2015 04:36 PM, Friedley, Andrew wrote:
> > Hi Mike,
> >
> > Have you contacted your admins, or the vendor that provided your True
> > Scale and/or PSM installation? E.g. Redhat, or Intel via
> > ibsupp...@intel.com ?  They are normally
> > the recommended path for True Scale support.
> >
> > That said, here's some things to look into:
> >
> > I think it's strange that you're hitting PSM linker problems while
> > compiling an app with Open MPI.  Normally only the PSM MTL is linked
> > directly against PSM.  Try verifying that nothing else is linking in
> > PSM. Also, it's possible your OMPI build is different from normal too.
> >
> > PSM has optional KNEM support that can be compiled in, but it's off by
> > default and I've never heard of it being used.  It's possible that
> > your PSM was changed or recompiled and part of your cluster upgrade,
> > and the installation isn't quite right.  It looks like your PSM was
> > compiled with KNEM support, but the KNEM library isn't being linked in
> > or is not being found.
> >
> > Andrew
> >
> > *From:* users [mailto:users-boun...@open-mpi.org] *On Behalf Of
> > *VanEss.Michael
> > *Sent:* Thursday, January 8, 2015 1:07 PM
> > *To:* us...@open-mpi.org
> > *Subject:* [OMPI users] libpsm_infinipath issues?
> >
> > Hello all,
> >
> > Our clusters were just upgraded to both a new version of PGI (14.9) as
> > well as openmpi (1.8.3).  Previous versions were 12.1 and 1.6
> > respectively, and those compiled and linked just fine.  The newest
> > versions are not linking my mpi applications at all.  Here's the problem:
> >
> > /opt/scyld/openmpi/1.8.3/pgi/bin/mpif90 -C chemcode_mpi.o
> > mod_mpichem.o plume_mpi.o amb2D_mpi.o fex.o jex.o use_tuv.o
> > run_lsode.o mod_amb.o utmgeo.o lsode.o newamb.o plume.o amb2D.o
> > solve.o mod_cdf.o calc_rates.o mod_tuv.o flux.o amb1D.o amb_com.o
> > newamb2D.o vs_ccode.o ck_errors.o newamb1D.o doChem.o
> mod_lsode.o
> > stode.o plume_com.o typeSizes.o netcdf.o mod_parameters.o
> mod_chem.o
> > runfile.o com_codes.o mod_SLAM.o mod_CPUFF.o calc_za.o
> mod_releases.o
> > mod_site.o mod_distance.o nuclear_dates.o mod_luparms.o deposition.o
> > diffusion.o getTJ.o mod_met.o met_data.o mod_h5tuv.o tuv10.o
> > mod_h5camx.o cmxamb.o \
> >
> >  -L/ensco/apps/cm/CMSOL/linux/zlib-1.2.1/lib
> > -L/ensco/apps/cm/CMSOL/linux/szip-2.1/lib -o \
> >
> >  chemcode_mpi  -L/opt/pgi/linux86-64/14.9/lib -lpgf90
> > -lpgf90rtl \
> >
> >  -L/ensco/apps/cm/CMSOL/linux/hdf5-1.8.3/lib -lhdf5_fortran -l
> > hdf5 -lz -lm
> >
> > /usr/lib64/libpsm_infinipath.so.1: undefined reference to `knem_put'
> >
> > /usr/lib64/libpsm_infinipath.so.1: undefined reference to
> `knem_open_device'
> >
> > /usr/lib64/libpsm_infinipath.so.1: undefined reference to `knem_get'
> >
> > /usr/lib64/libpsm_infinipath.so.1: undefined reference to
> > `knem_register_region'
> >
> > I've searched the net for any information on this and nothing has
> > seemed to help.  I'm fairly confident that all my variables and paths
> > to the new software is correct.
> >
> > Any ideas?
> >
> > Regards,
> >
> > Mike
> >
> > --
> > --
> >
> > The information contained in this email message is intended only for
> > the use of the individual(s) to whom it is addressed and may contain
> > information that is privileged and sensitive. If you are not the
> > intended recipient, or otherwise have received this communication in
> > error, please notify the sender immediately by email at the above
> > referenced address and note that any further dissemination,
> > distribution or copying of this communication is strictly prohibited.
> >
> > The U.S. Export Control Laws regulate the export and re-export of
> > technology originating in the United States. This includes the
> > electronic transmission of information and software to foreign
> > countries and to certain foreign nationals. Recipient agrees to abide
> > by these laws and their regulations -- including the U.S. Department
> > of Commerce Export Administration Regulations and the U.S. Department
> > of State International Traffic in Arms Regulations -- and not to
> > transfer, by electronic transmission or otherwise, any content derived
> > from this email to either a foreign national or a foreign

[OMPI users] Determine IB transport type of OpenMPI job

[Sasso, John (GE Power & Water, Non-GE)]

For a multi-node job using OpenMPI 1.6.5 over InfiniBand where the OFED library 
is used, is there a way to tell what IB transport type is being used (RC, UC, 
UD, etc)?

---john

[OMPI users] Issue with requests in openmpi?

[Moritz Hanke]

Hello,

When running the attached program with 128 processes I get the following 
errors for most runs:


_openmpi/1.8.4 with --enable-debug --enable-mem-debug with NAG 6.0:_

opal_list_remove_item - the item 0x195d820 is not on the list 0x2b05da5101e0
connect/btl_openib_connect_udcm.c:2132: udcm_send_timeout: Assertion 
`((0xdeafbeedULL << 32) + 0xdeafbeedULL) == ((opal_object_t *) 
(msg))->obj_magic_id' failed.


_openmpi/1.8.1 with NAG 6.0:_

deadlock

_openmpi/1.8.4 with Intel15 (also with NAG 6.0):_

Program terminated with signal 11, Segmentation fault.
#0  0x0070a885 in opal_memory_ptmalloc2_int_malloc ()
(gdb) bt
#0  0x0070a885 in opal_memory_ptmalloc2_int_malloc ()
#1  0x0070a05f in opal_memory_linux_memalign_hook ()
#2  0x004afc29 in ompi_free_list_grow ()
#3  0x00535fde in match_one ()
#4  0x00533e16 in mca_pml_ob1_recv_frag_callback_match ()
#5  0x00431e92 in btl_openib_handle_incoming ()
#6  0x00431413 in btl_openib_component_progress ()
#7  0x006862f6 in opal_progress ()
#8  0x0041f222 in ompi_request_default_test_any ()
#9  0x00428ca9 in PMPI_Testany ()
#10 0x0040775f in get_free_send_request_handle ()
at ***:34
#11 0x00407819 in isend (send_buffer=0x7fffe9672c14, count=1, 
dest=70, tag=1)

at ***:52
#12 0x00407b82 in main () at ***:112

or

Program terminated with signal 7, Bus error.
#0  0x0070a89a in opal_memory_ptmalloc2_int_malloc ()
(gdb) bt
#0  0x0070a89a in opal_memory_ptmalloc2_int_malloc ()
#1  0x0070a05f in opal_memory_linux_memalign_hook ()
#2  0x004afc29 in ompi_free_list_grow ()
#3  0x00535fde in match_one ()
#4  0x00533e16 in mca_pml_ob1_recv_frag_callback_match ()
#5  0x00431e92 in btl_openib_handle_incoming ()
#6  0x00431413 in btl_openib_component_progress ()
#7  0x006862f6 in opal_progress ()
#8  0x0041f222 in ompi_request_default_test_any ()
#9  0x00428ca9 in PMPI_Testany ()
#10 0x0040775f in get_free_send_request_handle ()
at ***:34
#11 0x00407819 in isend (send_buffer=0x7fffabba7b64, count=1, 
dest=7, tag=1)

at ***:52
#12 0x00407b82 in main () at ***:112

or

Program terminated with signal 11, Segmentation fault.
#0  0x0070a951 in opal_memory_ptmalloc2_int_malloc ()
(gdb) bt
#0  0x0070a951 in opal_memory_ptmalloc2_int_malloc ()
#1  0x0070a05f in opal_memory_linux_memalign_hook ()
#2  0x2b9f7b05b248 in mlx4_create_ah (pd=0x1212680, attr=0xb26988) 
at src/verbs.c:728
#3  0x2b9f70eafaed in __ibv_create_ah (pd=0xb26640, attr=0xb26988) 
at src/verbs.c:508

#4  0x0043c9df in udcm_module_start_connect ()
#5  0x00445bcd in mca_btl_openib_endpoint_send ()
#6  0x00546050 in mca_pml_ob1_send_request_start_copy ()
#7  0x005310e4 in mca_pml_ob1_isend ()
#8  0x00428689 in PMPI_Isend ()
#9  0x00407876 in isend (send_buffer=0x7fff0e12cd94, count=1, 
dest=82, tag=0)

at ***:54
#10 0x00407aae in main () at ***:95

_ openmpi/1.6.5 with NAG 5.3__:_

no error

With less processes the program runs fine most of the time. It looks 
like an issue with requests in Open MPI. Can someone confirm this?

(I am running my tests on an Intel Xeon cluster.)

Thanks!
Moritz
#include 
#include 
#include 

// send stuff
static MPI_Request * isend_requests = NULL;
static int num_isend_requests;

//taken from http://beige.ucs.indiana.edu/I590/node85.html
static void mpi_err_handler(int error_code) {

   if (error_code != MPI_SUCCESS) {

  char error_string[1024];
  int length_of_error_string, error_class;
  int global_rank; //!< rank of process in MPI_COMM_WORLD

  MPI_Comm_rank(MPI_COMM_WORLD, &global_rank);
  MPI_Error_class(error_code, &error_class);
  MPI_Error_string(error_class, error_string, &length_of_error_string);
  fprintf(stderr, "%3d: %s\n", global_rank, error_string);
  MPI_Error_string(error_code, error_string, &length_of_error_string);
  fprintf(stderr, "%3d: %s\n", global_rank, error_string);
  MPI_Abort(MPI_COMM_WORLD, error_code);
   }
}

static int get_free_send_request_handle () {

  int flag, idx;
  MPI_Status lstatus;

  // test whether any send request has been fulfilled
  mpi_err_handler(MPI_Testany(num_isend_requests, isend_requests, &idx, &flag,
  &lstatus));
  if (flag && idx != MPI_UNDEFINED) return idx;

  // look for free send request
  for (int i = 0; i < num_isend_requests; i++)
if (isend_requests[i] == MPI_REQUEST_NULL)
  return i;

  fputs("ERROR: Did not find free request handle in "
"get_free_send_request_handle.", stderr);
  exit(EXIT_FAILURE);

  return -1;
}

static void isend(void const * send_buffer, int count, int dest, int tag) {

  int request_idx = get_free_send_request_handle();

  mpi_err_handler(MPI_Isend(send_buffer, count, MPI_INT, dest, tag,
 

Re: [OMPI users] send and receive vectors + variable length

[Dave Goodell (dgoodell)]

On Jan 9, 2015, at 7:46 AM, Jeff Squyres (jsquyres)  wrote:

> Yes, I know examples 3.8/3.9 are blocking examples.
> 
> But it's morally the same as:
> 
> MPI_WAITALL(send_requests...)
> MPI_WAITALL(recv_requests...)
> 
> Strictly speaking, that can deadlock, too.  
> 
> It reality, it has far less chance of deadlocking than examples 3.8 and 3.9 
> (because you're likely within the general progression engine, and the 
> implementation will progress both the send and receive requests while in the 
> first WAITALL).  
> 
> But still, it would be valid for an implementation to *only* progress the 
> send requests -- and NOT the receive requests -- while in the first WAITALL.  
> Which makes it functionally equivalent to examples 3.8/3.9.

That's not true.  The implementation is required to make progress on all 
outstanding requests (assuming they can be progressed).  The following should 
not deadlock:

✂
for (...)  MPI_Isend(...)
for (...)  MPI_Irecv(...)
MPI_Waitall(send_requests...)
MPI_Waitall(recv_requests...)
✂

-Dave

Re: [OMPI users] send and receive vectors + variable length

[George Bosilca]

I totally agree with Dave here. Moreover, based on the logic exposed by
Jeff, there is no right solution because if one choose to first wait on the
receive requests this  also leads to a deadlock as the send requests might
not be progressed.

As a side note, posting the receive requests first minimize the potential
for unexpected requests.

  George.


On Fri, Jan 9, 2015 at 12:31 PM, Dave Goodell (dgoodell)  wrote:

> On Jan 9, 2015, at 7:46 AM, Jeff Squyres (jsquyres) 
> wrote:
>
> > Yes, I know examples 3.8/3.9 are blocking examples.
> >
> > But it's morally the same as:
> >
> > MPI_WAITALL(send_requests...)
> > MPI_WAITALL(recv_requests...)
> >
> > Strictly speaking, that can deadlock, too.
> >
> > It reality, it has far less chance of deadlocking than examples 3.8 and
> 3.9 (because you're likely within the general progression engine, and the
> implementation will progress both the send and receive requests while in
> the first WAITALL).
> >
> > But still, it would be valid for an implementation to *only* progress
> the send requests -- and NOT the receive requests -- while in the first
> WAITALL.  Which makes it functionally equivalent to examples 3.8/3.9.
>
> That's not true.  The implementation is required to make progress on all
> outstanding requests (assuming they can be progressed).  The following
> should not deadlock:
>
> ✂
> for (...)  MPI_Isend(...)
> for (...)  MPI_Irecv(...)
> MPI_Waitall(send_requests...)
> MPI_Waitall(recv_requests...)
> ✂
>
> -Dave
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26154.php

Re: [OMPI users] send and receive vectors + variable length

[Diego Avesani]

Dear Jeff, Dear George, Dear Dave, Dear all,

so, is it correct to use *MPI_Waitall *?
Is my program ok now? Do you see other problems?

Thanks again

Diego


On 9 January 2015 at 18:39, George Bosilca  wrote:

> I totally agree with Dave here. Moreover, based on the logic exposed by
> Jeff, there is no right solution because if one choose to first wait on the
> receive requests this  also leads to a deadlock as the send requests might
> not be progressed.
>
> As a side note, posting the receive requests first minimize the potential
> for unexpected requests.
>
>   George.
>
>
> On Fri, Jan 9, 2015 at 12:31 PM, Dave Goodell (dgoodell) <
> dgood...@cisco.com> wrote:
>
>> On Jan 9, 2015, at 7:46 AM, Jeff Squyres (jsquyres) 
>> wrote:
>>
>> > Yes, I know examples 3.8/3.9 are blocking examples.
>> >
>> > But it's morally the same as:
>> >
>> > MPI_WAITALL(send_requests...)
>> > MPI_WAITALL(recv_requests...)
>> >
>> > Strictly speaking, that can deadlock, too.
>> >
>> > It reality, it has far less chance of deadlocking than examples 3.8 and
>> 3.9 (because you're likely within the general progression engine, and the
>> implementation will progress both the send and receive requests while in
>> the first WAITALL).
>> >
>> > But still, it would be valid for an implementation to *only* progress
>> the send requests -- and NOT the receive requests -- while in the first
>> WAITALL.  Which makes it functionally equivalent to examples 3.8/3.9.
>>
>> That's not true.  The implementation is required to make progress on all
>> outstanding requests (assuming they can be progressed).  The following
>> should not deadlock:
>>
>> ✂
>> for (...)  MPI_Isend(...)
>> for (...)  MPI_Irecv(...)
>> MPI_Waitall(send_requests...)
>> MPI_Waitall(recv_requests...)
>> ✂
>>
>> -Dave
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/01/26154.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26155.php
>

Re: [OMPI users] send and receive vectors + variable length

[Jeff Squyres (jsquyres)]

On Jan 9, 2015, at 12:39 PM, George Bosilca  wrote:

> I totally agree with Dave here. Moreover, based on the logic exposed by Jeff, 
> there is no right solution because if one choose to first wait on the receive 
> requests this  also leads to a deadlock as the send requests might not be 
> progressed.

Nah, you could MPI_WAITALL on *all* the requests.  :-)

But regardless: Dave and I chatted on the phone and he convinced me that you 
guys (Dave+George) are correct.  That being said, the concept fairness is not 
defined in MPI, such that a given implementation's optimization choices *could* 
make subtle differences in fairness of progression of requests listed in the 
WAITALL vs. requests not listed in the WAITALL.  So YMMV with regards to 
performance portability.

But regardless of even that: like I said before, it's likely to always work in 
reality.  :-)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] send and receive vectors + variable length

[Diego Avesani]

What does it mean "YMMV"?



On 9 January 2015 at 19:44, Jeff Squyres (jsquyres) 
wrote:

> YMMV




Diego

Re: [OMPI users] send and receive vectors + variable length

[Jeff Squyres (jsquyres)]

On Jan 9, 2015, at 1:54 PM, Diego Avesani  wrote:

> What does it mean "YMMV"?

http://netforbeginners.about.com/od/xyz/f/What-Is-YMMV.htm

:-)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] send and receive vectors + variable length

[Diego Avesani]

Dear all,
thanks a lot,
really Thanks a lot

Diego


On 9 January 2015 at 19:56, Jeff Squyres (jsquyres) 
wrote:

> On Jan 9, 2015, at 1:54 PM, Diego Avesani  wrote:
>
> > What does it mean "YMMV"?
>
> http://netforbeginners.about.com/od/xyz/f/What-Is-YMMV.htm
>
> :-)
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26159.php
>

Re: [OMPI users] Determine IB transport type of OpenMPI job

[Joshua Ladd]

Open MPI's openib BTL only supports RC transport. 

Best,

Josh

Sent from my iPhone

> On Jan 9, 2015, at 9:03 AM, "Sasso, John (GE Power & Water, Non-GE)" 
>  wrote:
> 
> For a multi-node job using OpenMPI 1.6.5 over InfiniBand where the OFED 
> library is used, is there a way to tell what IB transport type is being used 
> (RC, UC, UD, etc)?
>  
> ---john
>  
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2015/01/26152.php

Re: [OMPI users] Determine IB transport type of OpenMPI job

[Sasso, John (GE Power & Water, Non-GE)]

Thank you!  Unless I missed something, I haven’t seen this in the OpenMPI FAQ.  
If it isn’t in there, it would be nice to include it.  Thanks!

--john

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Joshua Ladd
Sent: Friday, January 09, 2015 3:22 PM
To: Open MPI Users
Subject: Re: [OMPI users] Determine IB transport type of OpenMPI job

Open MPI's openib BTL only supports RC transport.

Best,

Josh

Sent from my iPhone

On Jan 9, 2015, at 9:03 AM, "Sasso, John (GE Power & Water, Non-GE)" 
mailto:john1.sa...@ge.com>> wrote:
For a multi-node job using OpenMPI 1.6.5 over InfiniBand where the OFED library 
is used, is there a way to tell what IB transport type is being used (RC, UC, 
UD, etc)?

---john

___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2015/01/26152.php