[OMPI users] Dynamic process allocation hangs

[L Lutret via users]

Hello all. Im started some test with Openmpi 4.0.1. I have two machines,
one local, the other remote. I have used ssh connection. Some basic test
(hello.c script) runs ok local and remote with mpirun. But I need to run a
script without mpirun and generate with spawn some processes. Here some
examples that what I get.


My hostfile:

cat hostfile


localhost slots=4

slave1 slots=4


If I set this:


MPI_Info_set( info, "add-hostfile", "hostfile" );

MPI_Info_set( info, "npernode", "3" );


And I run 6 processes (i.e. MPI_Comm_spawn() receives 6 procceses to run):


 ./dyamic.o


Its Runs Ok: 4 procceses local and 3 remote


Now, If I set (without add-hostfile and npernode):


  MPI_Info_set( info, "add-host", "slave1,slave1,slave1,slave1" );


And I run 4 processes... its hangs, but I can see with Top one running
processes on local and 4 on remote (slave1), that I think Its ok however.
After a while It throws this:


“A request has timed out and will therefore fail:


Operation: LOOKUP: orted/pmix/pmix_server_pub.c:345


Your job may terminate as a result of this problem. You may want to

adjust the MCA parameter pmix_server_max_wait and try again. If this

occurred during a connect/accept operation, you can adjust that time

using the pmix_base_exchange_timeout parameter.

--

[master:22881] *** An error occurred in MPI_Comm_spawn

[master:22881] *** reported by process [63766529,0]

[master:22881] *** on communicator MPI_COMM_WORLD

[master:22881] *** MPI_ERR_UNKNOWN: unknown error

[master:22881] *** MPI_ERRORS_ARE_FATAL (processes in this communicator
will now abort,

[master:22881] *** and potentially your MPI job)”


I watch with Top now and there are not any processes running.

I really need this type of allocation. Any help It will be very, very
appreciated.
Thanks in advance.

[OMPI users] Program hangs when MPI_Bcast is called rapidly

[Garrett, Charles via users]

I have a problem where MPI_Bcast hangs when called rapidly over and over again. 
 This problem manifests itself on our new cluster, but not on our older one.  
The new cluster has Cascade Lake processors.  Each node contains 2 sockets with 
18 cores per socket.  Cluster size is 128 nodes with an EDR infiniband network. 
 Below is a reproducer for the issue.  The hanging problem occurs in both the 
Zoltan partitioning software from DOE and Fun3d from NASA.

The reproducer fails with openmpi-3.1.4 and gcc-9.2.  The software mentioned 
above has failed with both gcc-9.2 and intel-19.2 with several versions of 
openmpi (from the 2 and 3 series).  One thing that seems to fix the problem is 
setting both:
   setenv OMPI_MCA_coll_sync_barrier_before 1
   setenv OMPI_MCA_coll_sync_barrier_after 1
Setting these to 10 still showed hanging problems in the software.  The 
reproducer hangs on our system when run with 30 nodes and 36 processes per 
node.  The make command is: mpicxx main.cpp -o test.x.  After salloc -n 1080, I 
run with mpirun ./test.x.

When the program hangs, I get on one of the allocated nodes and run gstack on 
one of the running PIDs to get a stack trace.  It always contains PMPI_Bcast.

Thanks in advance for any advice on the problem.
Kris Garrett


#include 
#include 

const int BUFFER_LEN = 2000;
const int OUTER_MAX = 200;

int main(int argc, char **argv)
{
   int mpi_size, my_rank;

   MPI_Init(&argc, &argv);
   MPI_Comm_size(MPI_COMM_WORLD, &mpi_size);
   MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

   for (int outer = 0; outer < OUTER_MAX; outer++)
   {
  for (int root = 0; root < mpi_size; root++)
  {
 int *buf;
 int len;

 if (my_rank == 0)
printf("%d\n", root);

 if (my_rank == root)
len = BUFFER_LEN;
 MPI_Bcast(&len, 1, MPI_INT, root, MPI_COMM_WORLD);
 buf = new int[len];
 if (my_rank == root)
 {
for (int i = 0; i < len; i++)
   buf[i] = 1.0;
 }
 MPI_Bcast(buf, len, MPI_INT, root, MPI_COMM_WORLD);
 delete[] buf;
  }
   }

   MPI_Finalize();
   return 0;
}