[OMPI users] Unknown breakdown (Transport retry count exceeded on mlx5_0:1/IB)

Hi, Good afternoon. I am using openmpi/4.0.3 on Compute Canada to do 3D flow simulation. It worked quite well for lower Reynolds number. However, after increasing it from 3600 to 9000, openmpi reported errors as shown below: [gra1288:149104:0:149104] ib_mlx5_log.c:132 Transport retry count exc

[OMPI users] MPI_ERR_TAG: invalid tag

Hi, Good morning. I am using openmpi/4.0.3 on Compute Canada to do 3D flow simulation. My grid size is Lx*Ly*Lz=700*169*500. It worked quite well for lower resolution. However, after increasing my resolution from Nx*Ny*Nz=64*109*62 to 256*131*192, openmpi reported errors as shown below: [gra5