[OMPI users] MPIRUN Error on Mac pro i7 laptop and linux desktop

[christophe petit]

hello,

i have a problem concerning the execution of a f90 program (explicitPar)
compiled with openmpi-1.4.2. I get nearly the same error on my debian
desktop ( AMD Phenom(tm) 9550 Quad-Core Processor) and my mac pro i7 laptop
:

on mac pro i7 :

$ mpiexec -np 2 explicitPar
[macbook-pro-de-fab.livebox.home:48805] *** An error occurred in
MPI_Cart_shift
[macbook-pro-de-fab.livebox.home:48805] *** on communicator MPI_COMM_WORLD
[macbook-pro-de-fab.livebox.home:48805] *** MPI_ERR_COMM: invalid
communicator
[macbook-pro-de-fab.livebox.home:48805] *** MPI_ERRORS_ARE_FATAL (your MPI
job will now abort)
--
mpiexec has exited due to process rank 1 with PID 48805 on
node macbook-pro-de-fab.livebox.home exiting without calling "finalize".
This may
have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).

---

on my debian desktop :

mpirun -np 2 explicitPar
[pablo:11665] *** An error occurred in MPI_Cart_shift
[pablo:11665] *** on communicator MPI_COMM_WORLD
[pablo:11665] *** MPI_ERR_COMM: invalid communicator
[pablo:11665] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--
mpirun has exited due to process rank 1 with PID 11665 on
node pablo exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--


I have installed openmpi-1.4.2 with the following options :

./configure --prefix=/usr/local/openmpi --enable-mpi-f77 --enable-mpi-f90

with exported variables on bash shell : FC=gfortran F90=gfortran
F77=gfortran CC=gcc CXX=g++

The  installation has been completed, the program compiles fine but i don't
understand what's wrong. I note that with a single processor ("mpirun -np 1
explicitPar"), execution works fine.

My debian desktop is a quad-core, so, theoretically, i can put up to 4 for
"np" parameter.
On my mac pro i7, i don't know how processors are there, but the "htop"
command makes appear 4 cores too.

Anyone has a solution ?

Regards.

Re: [OMPI users] MPIRUN Error on Mac pro i7 laptop and linux desktop

[christophe petit]

(*,1002) t,result
!
   if ((result.gt.epsilon).and.(step.lt.maxStep)) goto 10
!
! UNTIL "Convergence"
!
   tend = MPI_Wtime()
   if (me.eq.0) then
 write(*,*)
 write(*,*) ' Convergence after ', step,' steps '
 write(*,*) '  Problem size  ',
size_x*x_domains*size_y*y_domains
 write(*,*) ' Wall Clock ', tend-tstart

!
! Print the solution at each point of the grid
!
 write(*,*)
 write(*,*) ' Computed solution '
 write(*,*)
 do 30, j=size_x+1,0,-1
write(*,1000)(x0(j,i),i=0,size_x+1)
 30  continue
   endif
!
  call MPI_FINALIZE(infompi)
!
  deallocate(x)
  deallocate(x0)
  deallocate(b)
!
! Formats available to display the computed values on the grid
!
1000  format(100(1x, f7.3))
1001  format(100(1x, e7.3))
1002   format(' At time ',E8.2,' Norm ', E8.2)

!
  stop
  end
!

--
2010/7/28 Jeff Squyres 

> According to the error message (especially since it's consistent across 2
> different platforms), it looks like you have an error in your application.
>  Open MPI says that you're using an invalid communicator when calling
> MPI_Cart_shift.  "Invalid" probably means that it's not a Cartesian
> communicator.
>
> You might want to double check the definition and requirements of the
> MPI_CART_SHIFT function (see the MPI_Cart_shift(3) man page).
>
>
>
> On Jul 28, 2010, at 12:28 PM, christophe petit wrote:
>
> > hello,
> >
> > i have a problem concerning the execution of a f90 program (explicitPar)
> compiled with openmpi-1.4.2. I get nearly the same error on my debian
> desktop ( AMD Phenom(tm) 9550 Quad-Core Processor) and my mac pro i7 laptop
> :
> >
> > on mac pro i7 :
> >
> > $ mpiexec -np 2 explicitPar
> > [macbook-pro-de-fab.livebox.home:48805] *** An error occurred in
> MPI_Cart_shift
> > [macbook-pro-de-fab.livebox.home:48805] *** on communicator
> MPI_COMM_WORLD
> > [macbook-pro-de-fab.livebox.home:48805] *** MPI_ERR_COMM: invalid
> communicator
> > [macbook-pro-de-fab.livebox.home:48805] *** MPI_ERRORS_ARE_FATAL (your
> MPI job will now abort)
> >
> --
> > mpiexec has exited due to process rank 1 with PID 48805 on
> > node macbook-pro-de-fab.livebox.home exiting without calling "finalize".
> This may
> > have caused other processes in the application to be
> > terminated by signals sent by mpiexec (as reported here).
> >
> >
> ---
> >
> > on my debian desktop :
> >
> > mpirun -np 2 explicitPar
> > [pablo:11665] *** An error occurred in MPI_Cart_shift
> > [pablo:11665] *** on communicator MPI_COMM_WORLD
> > [pablo:11665] *** MPI_ERR_COMM: invalid communicator
> > [pablo:11665] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
> >
> --
> > mpirun has exited due to process rank 1 with PID 11665 on
> > node pablo exiting without calling "finalize". This may
> > have caused other processes in the application to be
> > terminated by signals sent by mpirun (as reported here).
> >
> --
> >
> >
> > I have installed openmpi-1.4.2 with the following options :
> >
> > ./configure --prefix=/usr/local/openmpi --enable-mpi-f77 --enable-mpi-f90
> >
> > with exported variables on bash shell : FC=gfortran F90=gfortran
> F77=gfortran CC=gcc CXX=g++
> >
> > The  installation has been completed, the program compiles fine but i
> don't understand what's wrong. I note that with a single processor ("mpirun
> -np 1 explicitPar"), execution works fine.
> >
> > My debian desktop is a quad-core, so, theoretically, i can put up to 4
> for "np" parameter.
> > On my mac pro i7, i don't know how processors are there, but the "htop"
> command makes appear 4 cores too.
> >
> > Anyone has a solution ?
> >
> > Regards.
> >
> >
> >
> >
> >
> >
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] try to understand heat equation 2D mpi version

[christophe petit]

Hello,

i'm studying the parallelized version of a solving 2D heat equation code in
order to understand cartesian topology and the famous "MPI_CART_SHIFT".
Here's my problem at this part of the code :


---
call MPI_INIT(infompi)
  comm = MPI_COMM_WORLD
  call MPI_COMM_SIZE(comm,nproc,infompi)
  call MPI_COMM_RANK(comm,me,infompi)
!

..


! Create 2D cartesian grid
  periods(:) = .false.

  ndims = 2
  dims(1)=x_domains
  dims(2)=y_domains
  CALL MPI_CART_CREATE(MPI_COMM_WORLD, ndims, dims, periods, &
reorganisation,comm2d,infompi)
!
! Identify neighbors
!
  NeighBor(:) = MPI_PROC_NULL
! Left/West and right/Est neigbors
  CALL MPI_CART_SHIFT(comm2d,0,1,NeighBor(W),NeighBor(E),infompi)

  print *,'rank=', me
  print *, 'here first mpi_cart_shift : neighbor(w)=',NeighBor(W)
  print *, 'here first mpi_cart_shift : neighbor(e)=',NeighBor(E)

...

---

with x_domains=y_domains=2

and i get at the execution :" mpirun -np 4 ./explicitPar"

 rank=   0
here first mpi_cart_shift : neighbor(w)=  -1
 here first  mpi_cart_shift : neighbor(e)=   2
rank=   3
 here first mpi_cart_shift : neighbor(w)=   1
 here first mpi_cart_shift : neighbor(e)=  -1
 rank=   2
 here first mpi_cart_shift : neighbor(w)=   0
 here first mpi_cart_shift : neighbor(e)=  -1
 rank=   1
 here first mpi_cart_shift : neighbor(w)=  -1
 here first mpi_cart_shift : neighbor(e)=   3

I saw that if the rank is out of the topology and wihtout periodicity, the
rank should be equal to MPI_UNDEFINED whis is assigned to -32766 in "mpif.h"
. So, why have i got the value "-1" ?
On my Macbook pro, i get the value "-2".

Thanks in advance.

[OMPI users] try to understand heat equation 2D mpi solving version

[christophe petit]

Hello,

i am still trying to understand the parallelized version of the heat
equation 2D solving that we saw at school. In order to explain my problem, i
need to list the main code :

  9   program heat
 10
!**
 11 !
 12 !   This program solves the heat equation on the unit square [0,1]x[0,1]
 13 !| du/dt - Delta(u) = 0
 14 !|  u/gamma = cste
 15 !   by implementing a explicit scheme.
 16 !   The discretization is done using a 5 point finite difference scheme
 17 !   and the domain is decomposed into sub-domains.
 18 !   The PDE is discretized using a 5 point finite difference scheme
 19 !   over a (x_dim+2)*(x_dim+2) grid including the end points
 20 !   correspond to the boundary points that are stored.
 21 !
 22 !   The data on the whole domain are stored in
 23 !   the following way :
 24 !
 25 !y
 26 !   
 27 !d  |  |
 28 !i  |  |
 29 !r  |  |
 30 !e  |  |
 31 !c  |  |
 32 !t  |  |
 33 !i  | x20  |
 34 !o /\   |  |
 35 !n  |   | x10  |
 36 !   |   |  |
 37 !   |   | x00  x01 x02 ... |
 38 !   |   
 39 !---> x direction  x(*,j)
 40 !
 41 !   The boundary conditions are stored in the following submatrices
 42 !
 43 !
 44 !x(1:x_dim, 0)  ---> left   temperature
 45 !x(1:x_dim, x_dim+1)---> right  temperature
 46 !x(0, 1:x_dim)  ---> toptemperature
 47 !x(x_dim+1, 1:x_dim)---> bottom temperature
 48 !
 49
!**
 50   implicit none
 51   include 'mpif.h'
 52 ! size of the discretization
 53   integer :: x_dim, nb_iter
 54   double precision, allocatable :: x(:,:),b(:,:),x0(:,:)
 55   double precision  :: dt, h, epsilon
 56   double precision  :: resLoc, result, t, tstart, tend
 57 !
 58   integer :: i,j
 59   integer :: step, maxStep
 60   integer :: size_x, size_y, me, x_domains,y_domains
 61   integer :: iconf(5), size_x_glo
 62   double precision conf(2)
 63 !
 64 ! MPI variables
 65   integer :: nproc, infompi, comm, comm2d, lda, ndims
 66   INTEGER, DIMENSION(2)  :: dims
 67   LOGICAL, DIMENSION(2)  :: periods
 68   LOGICAL, PARAMETER :: reorganisation = .false.
 69   integer :: row_type
 70   integer, parameter :: nbvi=4
 71   integer, parameter :: S=1, E=2, N=3, W=4
 72   integer, dimension(4) :: neighBor
 73
 74 !
 75   intrinsic abs
 76 !
 77 !
 78   call MPI_INIT(infompi)
 79   comm = MPI_COMM_WORLD
 80   call MPI_COMM_SIZE(comm,nproc,infompi)
 81   call MPI_COMM_RANK(comm,me,infompi)
 82 !
 83 !
 84   if (me.eq.0) then
 85   call readparam(iconf, conf)
 86   endif
 87   call MPI_BCAST(iconf,5,MPI_INTEGER,0,comm,infompi)
 88   call MPI_BCAST(conf,2,MPI_DOUBLE_PRECISION,0,comm,infompi)
 89 !
 90   size_x= iconf(1)
 91   size_y= iconf(1)
 92   x_domains = iconf(3)
 93   y_domains = iconf(4)
 94   maxStep   = iconf(5)
 95   dt= conf(1)
 96   epsilon   = conf(2)
 97 !
 98   size_x_glo = x_domains*size_x+2
 99   h  = 1.0d0/dble(size_x_glo)
100   dt = 0.25*h*h
101 !
102 !
103   lda = size_y+2
104   allocate(x(0:size_y+1,0:size_x+1))
105   allocate(x0(0:size_y+1,0:size_x+1))
106   allocate(b(0:size_y+1,0:size_x+1))
107 !
108 ! Create 2D cartesian grid
109   periods(:) = .false.
110
111   ndims = 2
112   dims(1)=x_domains
113   dims(2)=y_domains
114   CALL MPI_CART_CREATE(MPI_COMM_WORLD, ndims, dims, periods, &
115 reorganisation,comm2d,infompi)
116 !
117 ! Identify neighbors
118 !
119   NeighBor(:) = MPI_PROC_NULL
120 ! Left/West and right/Est neigbors
121   CALL MPI_CART_SHIFT(comm2d,0,1,NeighBor(W),NeighBor(E),infompi)
122
123   print *,'mpi_proc_null=', MPI_PROC_NULL
124   print *,'rang=', me
125   print *, 'ici premier mpi_cart_shift : neighbor(w)=',NeighBor(W)
126   print *, 'ici premier mpi_cart_shift : neighbor(e)=',NeighBor(E)
127
128 ! Bottom/South and Upper/North neigbors
129   CALL MPI_CART_SHIFT(comm2d,1,1,NeighBor(S),NeighBor(N),infompi)
130
131
132   print *, 'ici deuxieme mpi_cart_shift : neighbor(s)=',NeighBor(S)
133   print *, 'ici deuxieme mpi_cart_shift : neighbor(n)=',NeighBor(N)
134
135
136
137 !
138 ! Create row data type to coimmunicate with South and North neighbors
139 !
140   CALL MPI_TYPE_VECTOR(size_x

[OMPI users] Need Help for understand heat equation 2D mpi solving version

[christophe petit]

Hello,

>
> i am still trying to understand the parallelized version of the heat
> equation 2D solving that we saw at school. In order to explain my problem, i
> need to list the main code :
>
>   9   program heat
>  10
> !**
>  11 !
>  12 !   This program solves the heat equation on the unit square
> [0,1]x[0,1]
>  13 !| du/dt - Delta(u) = 0
>  14 !|  u/gamma = cste
>  15 !   by implementing a explicit scheme.
>  16 !   The discretization is done using a 5 point finite difference scheme
>  17 !   and the domain is decomposed into sub-domains.
>  18 !   The PDE is discretized using a 5 point finite difference scheme
>  19 !   over a (x_dim+2)*(x_dim+2) grid including the end points
>  20 !   correspond to the boundary points that are stored.
>  21 !
>  22 !   The data on the whole domain are stored in
>  23 !   the following way :
>  24 !
>  25 !y
>  26 !   
>  27 !d  |  |
>  28 !i  |  |
>  29 !r  |  |
>  30 !e  |  |
>  31 !c  |  |
>  32 !t  |  |
>  33 !i  | x20  |
>  34 !o /\   |  |
>  35 !n  |   | x10  |
>  36 !   |   |  |
>  37 !   |   | x00  x01 x02 ... |
>  38 !   |   
>  39 !---> x direction  x(*,j)
>  40 !
>  41 !   The boundary conditions are stored in the following submatrices
>  42 !
>  43 !
>  44 !x(1:x_dim, 0)  ---> left   temperature
>  45 !x(1:x_dim, x_dim+1)---> right  temperature
>  46 !x(0, 1:x_dim)  ---> toptemperature
>  47 !x(x_dim+1, 1:x_dim)---> bottom temperature
>  48 !
>  49
> !**
>  50   implicit none
>  51   include 'mpif.h'
>  52 ! size of the discretization
>  53   integer :: x_dim, nb_iter
>  54   double precision, allocatable :: x(:,:),b(:,:),x0(:,:)
>  55   double precision  :: dt, h, epsilon
>  56   double precision  :: resLoc, result, t, tstart, tend
>  57 !
>  58   integer :: i,j
>  59   integer :: step, maxStep
>  60   integer :: size_x, size_y, me, x_domains,y_domains
>  61   integer :: iconf(5), size_x_glo
>  62   double precision conf(2)
>  63 !
>  64 ! MPI variables
>  65   integer :: nproc, infompi, comm, comm2d, lda, ndims
>  66   INTEGER, DIMENSION(2)  :: dims
>  67   LOGICAL, DIMENSION(2)  :: periods
>  68   LOGICAL, PARAMETER :: reorganisation = .false.
>  69   integer :: row_type
>  70   integer, parameter :: nbvi=4
>  71   integer, parameter :: S=1, E=2, N=3, W=4
>  72   integer, dimension(4) :: neighBor
>  73
>  74 !
>  75   intrinsic abs
>  76 !
>  77 !
>  78   call MPI_INIT(infompi)
>  79   comm = MPI_COMM_WORLD
>  80   call MPI_COMM_SIZE(comm,nproc,infompi)
>  81   call MPI_COMM_RANK(comm,me,infompi)
>  82 !
>  83 !
>  84   if (me.eq.0) then
>  85   call readparam(iconf, conf)
>  86   endif
>  87   call MPI_BCAST(iconf,5,MPI_INTEGER,0,comm,infompi)
>  88   call MPI_BCAST(conf,2,MPI_DOUBLE_PRECISION,0,comm,infompi)
>  89 !
>  90   size_x= iconf(1)
>  91   size_y= iconf(1)
>  92   x_domains = iconf(3)
>  93   y_domains = iconf(4)
>  94   maxStep   = iconf(5)
>  95   dt= conf(1)
>  96   epsilon   = conf(2)
>  97 !
>  98   size_x_glo = x_domains*size_x+2
>  99   h  = 1.0d0/dble(size_x_glo)
> 100   dt = 0.25*h*h
> 101 !
> 102 !
> 103   lda = size_y+2
> 104   allocate(x(0:size_y+1,0:size_x+1))
> 105   allocate(x0(0:size_y+1,0:size_x+1))
> 106   allocate(b(0:size_y+1,0:size_x+1))
> 107 !
> 108 ! Create 2D cartesian grid
> 109   periods(:) = .false.
> 110
> 111   ndims = 2
> 112   dims(1)=x_domains
> 113   dims(2)=y_domains
> 114   CALL MPI_CART_CREATE(MPI_COMM_WORLD, ndims, dims, periods, &
> 115 reorganisation,comm2d,infompi)
> 116 !
> 117 ! Identify neighbors
> 118 !
> 119   NeighBor(:) = MPI_PROC_NULL
> 120 ! Left/West and right/Est neigbors
> 121   CALL MPI_CART_SHIFT(comm2d,0,1,NeighBor(W),NeighBor(E),infompi)
> 122
> 123   print *,'mpi_proc_null=', MPI_PROC_NULL
> 124   print *,'rang=', me
> 125   print *, 'ici premier mpi_cart_shift : neighbor(w)=',NeighBor(W)
> 126   print *, 'ici premier mpi_cart_shift : neighbor(e)=',NeighBor(E)
> 127
> 128 ! Bottom/South and Upper/North neigbors
> 129   CALL MPI_CART_SHIFT(comm2d,1,1,NeighBor(S),NeighBor(N),infompi)
> 130
> 131
> 132   print *, '

[OMPI users] [SPAM:### 84%]

[christophe petit]

http://proappreviews.com/wp-content/themes/twentyten/friends.html

[OMPI users] [SPAM:### 84%]

[christophe petit]

http://www.pimp2.com/modules/mod_osdonate/life.html

[OMPI users] need help for a mpi 2d heat equation solving code

[christophe petit]

Hello,

i need help regarding a mpi program which solves the 2d heat equation. I
have rewritten the original code in my own way,
in oder to understand well the parallelization.

I will initially make it work with 4 processors (nproc=4)  with a 2D domain of
100 points, that is to say 10 on the x axis (size_x=10)
and 10 on the y axis (size_y=10). The 2D is divided into four (x_domains=2
and y_domains=2).

The array of values ​​( "x0 in the code ) has a dimension of 8 * 8, so each
processor works on 4 * 4 arrays.

The current process rank is stored in "me".  I calculate the coordinates of
the domain interval for each worker :

xdeb(me)ftp://toulouse-immobilier.bz/pub/param>file.

There are a total of three main files : the main program
explicitPar.f90which
does initialization,
calls in the main loop : the explitUtil solving routine in
explUtil.f90and updates
the
neighbors of the current process with updateBound routine in
updateBound.f90.

Everything seems ok except the "updateBound" routine :  I have a problem
with the indexes of row and columns in the communication between
North Neighbors, South Neighbors, West and East Neighbors.

For example, I have :


! Send my boundary to North and receive from South

CALL MPI_SENDRECV(x(ydeb(me),xdeb(me)), 1, row_type ,neighBor(N),flag, &
 x(yfin(me),xdeb(me)), 1, row_type,neighbor(S),flag, &
 comm2d, status, infompi)


For 4 processors and me=0, i have :

xdeb(0)=1
xfin(0)=4
ydeb(0)=5
yfin(0 )=8

So I send to the North neighbor the upper row indexed by
x(ydeb(me),xdeb(me)). But I should have ghost cells for
the communication in this case for calculting the next values of "x0" array
for each worker. Actually I need the
boundaries values for each worker ( with 4*4 size) but I think I have to
work on a 5*5 size for this calculation in order to
have ghost cells on the edges of 4*4 worker cells.

You can compile this code by adapting the
makefilewith :" $ make
explicitPar "
and execute it in my case with : "$ mpirun -n 4 explicitPar "


Anyone could see what's wrong with my code ? Have I got to put a 12*12 size
for the global domain ? it would allow to have 5*5 worker cells with 2 more
for the boundary condition (constant), so a total equal to 12 for size_x and
size_y ( 8+2+2).

Note that the edges of the domain remain equal to 10 as it's expected.

Any help would be really appreciated.

Thanks in advance.

[OMPI users] Problem on 3D contiguous array with MPI_TYPE_VECTOR

[christophe petit]

Hello,

I try to use MPI for solving the Fourier equation in 3D. In the code, I
have the following parameters :

number of domains on Ox : x_domains
number of domains on Oy : y_domains
number of domains on Oz : z_domains

size of grid on Ox: size_x
size of grid on Oy: size_y
size of grid on Oz: size_z

elements per Ox domain : xcell=(size_x/x_domains)
elements per Oy domain : ycell=(size_y/y_domains)
elements per Oz domain : zcell=(size_z/z_domains)

Total sizes of the 3D grid (including the ghost cells) :

size_tot_x=size_x+2*x_domains+2
size_tot_y=size_y+2*y_domains+2
size_tot_z=size_z+2*z_domains+2



I have determined the coordinates of corners of each box as function of the
rank process "me" :

I have "xs(me)" (x start), "xe(me)" (x end), "ys(me)", "ye(me)", "zs(me)"
and "ze(me)".

Moreover, I have defined 3 types of matrix ( Oxz plan, Oxy and 0yz ) for
the intercommunication between the processes. I am not sure about *the
distance between two elements* in the following definition of the *matrix
type* :
--
! Creation of "Matrix" type :

   call MPI_TYPE_VECTOR (nb_columns, nb_rows,* step_between_two_elements* ,
MPI_DOUBLE_PRECISION ,* matrix_type *,code)
---

Here are the part of the code where I have defined the 3 types of Matrix :

---
! Create matrix data type to communicate on horizontal Oxz plan

  CALL MPI_TYPE_VECTOR( xcell, zcell, *size_tot_y*,
MPI_DOUBLE_PRECISION &
 , matrix_type_oxz, infompi)
  CALL MPI_TYPE_COMMIT(matrix_type_oxz, infompi)
---

---
! Create matrix data type to communicate on vertical Oxy plan

  CALL MPI_TYPE_VECTOR( xcell, ycell, *size_tot_z*,
MPI_DOUBLE_PRECISION &
 , matrix_type_oxy, infompi)
  CALL MPI_TYPE_COMMIT(matrix_type_oxy, infompi)
---

---
! Create matrix data type to communicate on vertical Oyz plan

  CALL MPI_TYPE_VECTOR( zcell, ycell, *size_tot_x*,
MPI_DOUBLE_PRECISION &
 , matrix_type_oyz, infompi)
  CALL MPI_TYPE_COMMIT(matrix_type_oyz, infompi)
---

As you can see, I have doubts about the contiguous distance (*size_tot_x,
size_tot_y, size_tot_z*) for the 3 types above.

For the 2D case, I use the row_type and the contiguous distance is
"size_tot_y" but for 3D, I don't know .

Could you help me please ?


Thanks.

[OMPI users] issue with 2 connected computers

[christophe petit]

Hello,

I try to launch an executable on 2 computers ( Debian 6.0 and MacOS snow
leopard ).

I have successfully installed on both Open-MPI 1.6 with the option
"--enable-heterogenous" and
setup passwordless ssh connection between the 2 computers.

My issue is that the distributed computing works fine but only when I
launch the
"mpirun -np 16 -hostfile hosts.txt program_exec" command from the MacOS
computer, not from Debian pc.

However, I have disabled firewall on MacOS with :

sysctl -w net.inet.ip.fw.enable=0

When I launched from Debian, I get :

*~/mpirun -np 16 -hostfile hosts.txt program_exec

[maco:01498] Error: unknown option "--daemonize"
Usage: orted [OPTION]...
Start an Open RTE Daemon

   --bootproxy Run as boot proxy for 
-d|--debug   Debug the OpenRTE
-d|--spinHave the orted spin until we can connect a debugger
 to it
   --debug-daemons   Enable debugging of OpenRTE daemons
   --debug-daemons-file  Enable debugging of OpenRTE daemons, storing output
 in files
   --gprreplicaRegistry contact information.
-h|--helpThis help message
   --mpi-call-yield 
 Have MPI (or similar) applications call yield when
 idle
   --name  Set the orte process name
   --no-daemonizeDon't daemonize into the background
   --nodename  Node name as specified by host/resource
 description.
   --ns-ndsset sds/nds component to use for daemon (normally
 not needed)
   --nsreplica Name service contact information.
   --num_procs Set the number of process in this job
   --persistent  Remain alive after the application process
 completes
   --report-uriReport this process' uri on indicated pipe
   --scope Set restrictions on who can connect to this
 universe
   --seedHost replicas for the core universe services
   --set-sid Direct the orted to separate from the current
 session
   --tmpdirSet the root for the session directory tree
   --universe  Set the universe name as
 username@hostname:universe_name for this
 application
   --vpid_startSet the starting vpid for this job
--
A daemon (pid 24370) died unexpectedly with status 251 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--
--
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--
*
LD_LIBRARY_PATH and PATH are correctly set on both ( Open-MPI is installed
on
"/usr/local/openmpi/" ).

Apparently, the problem comes from the Debian pc ...

If anyone sees what's wrong ?

Thanks.

[OMPI users] random problems with a ring communication example

[christophe petit]

Hello,

I followed a simple MPI example to do a ring communication.

Here's the figure that illustrates this example with 7 processes :

http://i.imgur.com/Wrd6acv.png

Here the code :

--
 program ring

 implicit none
 include 'mpif.h'

 integer, dimension( MPI_STATUS_SIZE ) :: status
 integer, parameter:: tag=100
 integer :: nb_procs, rank, value, &
num_proc_previous,num_proc_next,code

 call MPI_INIT (code)
 call MPI_COMM_SIZE ( MPI_COMM_WORLD ,nb_procs,code)
 call MPI_COMM_RANK ( MPI_COMM_WORLD ,rank,code)

 num_proc_next=mod(rank+1,nb_procs)
 num_proc_previous=mod(nb_procs+rank-1,nb_procs)

 if (rank == 0) then
call MPI_SEND (1000,1, MPI_INTEGER ,num_proc_next,tag, &
   MPI_COMM_WORLD ,code)
call MPI_RECV (value,1, MPI_INTEGER ,num_proc_previous,tag, &
   MPI_COMM_WORLD ,status,code)
 else
call MPI_RECV (value,1, MPI_INTEGER ,num_proc_previous,tag, &
   MPI_COMM_WORLD ,status,code)
call MPI_SEND (rank+1000,1, MPI_INTEGER ,num_proc_next,tag, &
   MPI_COMM_WORLD ,code)
 end if
 print *,'Me, process ',rank,', I have received ',value,' from process
',num_proc_previous

 call MPI_FINALIZE (code)
end program ring

--

At the execution, I expect to always have :

Me, process1 , I have received 1000  from
process0
 Me, process2 , I have received 1001  from
process1
 Me, process3 , I have received 1002  from
process2
 Me, process4 , I have received 1003  from
process3
 Me, process5 , I have received 1004  from
process4
 Me, process6 , I have received 1005  from
process5
 Me, process0 , I have received 1006  from
process6

But sometimes, I have the reception of process 0 from process 6 which is
not the last reception, like this :

 Me, process1 , I have received 1000  from
process0
 Me, process2 , I have received 1001  from
process1
 Me, process3 , I have received 1002  from
process2
 Me, process4 , I have received 1003  from
process3
 Me, process5 , I have received 1004  from
process4
 Me, process0 , I have received 1006  from
process6
 Me, process6 , I have received 1005  from
process5

where reception of process 0 from process 6 happens before the reception of
process 6 from process 5

or like on this result :

 Me, process1 , I have received 1000  from
process0
 Me, process2 , I have received 1001  from
process1
 Me, process3 , I have received 1002  from
process2
 Me, process4 , I have received 1003  from
process3
 Me, process0 , I have received 1006  from
process6
 Me, process5 , I have received 1004  from
process4
 Me, process6 , I have received 1005  from
process5

where process 0 receives between the reception of process 4 and 5.

How can we explain this strange result ? I thought that standard use of
MPI_SEND and MPI_RECV were blocking by default and,
with this result, it seems to be not blocking.

I tested this example on Debian 7.0 with open-mpi package.

Thanks for your help

Re: [OMPI users] random problems with a ring communication example

[christophe petit]

Ok, so from what you say, on a "execution system" point view, the ring
communication is well achieved (i.e respecting the good order with, in last
position, rank0 which receives from rank 6) but the stdout doesn't reflect
what really happened, does it ?

Is there a way to make stdout respect the expected order ?

Thanks


2014-03-16 0:42 GMT+01:00 Ralph Castain :

> The explanation is simple: there is no rule about ordering of stdout. So
> even though your rank0 may receive its MPI message last, its stdout may
> well be printed before one generated on a remote node. Reason is that rank
> 0 may well be local to mpirun, and thus the stdout can be handled
> immediately. However, your rank6 may well be on a remote node, and that
> daemon has to forward the stdout to mpirun for printing.
>
> Like I said - no guarantee about ordering of stdout.
>
>
> On Mar 15, 2014, at 2:43 PM, christophe petit <
> christophe.peti...@gmail.com> wrote:
>
> Hello,
>
> I followed a simple MPI example to do a ring communication.
>
> Here's the figure that illustrates this example with 7 processes :
>
> http://i.imgur.com/Wrd6acv.png
>
> Here the code :
>
>
> --
>  program ring
>
>  implicit none
>  include 'mpif.h'
>
>  integer, dimension( MPI_STATUS_SIZE ) :: status
>  integer, parameter:: tag=100
>  integer :: nb_procs, rank, value, &
> num_proc_previous,num_proc_next,code
>
>  call MPI_INIT (code)
>  call MPI_COMM_SIZE ( MPI_COMM_WORLD ,nb_procs,code)
>  call MPI_COMM_RANK ( MPI_COMM_WORLD ,rank,code)
>
>  num_proc_next=mod(rank+1,nb_procs)
>  num_proc_previous=mod(nb_procs+rank-1,nb_procs)
>
>  if (rank == 0) then
> call MPI_SEND (1000,1, MPI_INTEGER ,num_proc_next,tag, &
>MPI_COMM_WORLD ,code)
> call MPI_RECV (value,1, MPI_INTEGER ,num_proc_previous,tag, &
>MPI_COMM_WORLD ,status,code)
>  else
> call MPI_RECV (value,1, MPI_INTEGER ,num_proc_previous,tag, &
>MPI_COMM_WORLD ,status,code)
> call MPI_SEND (rank+1000,1, MPI_INTEGER ,num_proc_next,tag, &
>MPI_COMM_WORLD ,code)
>  end if
>  print *,'Me, process ',rank,', I have received ',value,' from process
> ',num_proc_previous
>
>  call MPI_FINALIZE (code)
> end program ring
>
>
> --
>
> At the execution, I expect to always have :
>
> Me, process1 , I have received 1000  from
> process0
>  Me, process2 , I have received 1001  from
> process1
>  Me, process3 , I have received 1002  from
> process2
>  Me, process4 , I have received 1003  from
> process3
>  Me, process5 , I have received 1004  from
> process4
>  Me, process6 , I have received 1005  from
> process5
>  Me, process0 , I have received 1006  from
> process6
>
> But sometimes, I have the reception of process 0 from process 6 which is
> not the last reception, like this :
>
>  Me, process1 , I have received 1000  from
> process0
>  Me, process2 , I have received 1001  from
> process1
>  Me, process3 , I have received 1002  from
> process2
>  Me, process4 , I have received 1003  from
> process3
>  Me, process5 , I have received 1004  from
> process4
>  Me, process0 , I have received 1006  from
> process6
>  Me, process6 , I have received 1005  from
> process5
>
> where reception of process 0 from process 6 happens before the reception
> of process 6 from process 5
>
> or like on this result :
>
>  Me, process1 , I have received 1000  from
> process0
>  Me, process2 , I have received 1001  from
> process1
>  Me, process3 , I have received 1002  from
> process2
>  Me, process4 , I have received 1003  from
> process3
>  Me, process0 , I have received 1006  from
> process6
>  Me, process5 , I have received 1004  from
> process 

[OMPI users] efficient strategy with temporary message copy

[christophe petit]

Hello,

I am studying the optimization strategy when the number of communication
functions in a code
is high.

My courses on MPI say two things for optimization which are contradictory :

1*) You have to use temporary message copy to allow non-blocking sending
and uncouple the sending and receiving

2*) Avoid using temporary message copy because the copy will add extra cost
on execution time.

And then, we are adviced to do :

- replace MPI_SEND by MPI_SSEND (synchroneous blocking sending) : it is
said that execution is divided by a factor 2

- use MPI_ISSEND and MPI_IRECV with MPI_WAIT function to synchronize
(synchroneous non-blocking sending) : it is said that execution is divided
by a factor 3

So what's the best optimization ? Do we have to use temporary message copy
or not and if yes, what's the case for ?

Thanks

Re: [OMPI users] efficient strategy with temporary message copy

[christophe petit]

Thanks Jeff, I understand better the different cases and how to choose as a
function of the situation


2014-03-17 16:31 GMT+01:00 Jeff Squyres (jsquyres) :

> On Mar 16, 2014, at 10:24 PM, christophe petit <
> christophe.peti...@gmail.com> wrote:
>
> > I am studying the optimization strategy when the number of communication
> functions in a codeis high.
> >
> > My courses on MPI say two things for optimization which are
> contradictory :
> >
> > 1*) You have to use temporary message copy to allow non-blocking sending
> and uncouple the sending and receiving
>
> There's a lot of schools of thought here, and the real answer is going to
> depend on your application.
>
> If the message is "short" (and the exact definition of "short" depends on
> your platform -- it varies depending on your CPU, your memory, your
> CPU/memory interconnect, ...etc.), then copying to a pre-allocated bounce
> buffer is typically a good idea.  That lets you keep using your "real"
> buffer and not have to wait until communication is done.
>
> For "long" messages, the equation is a bit different.  If "long" isn't
> "enormous", you might be able to have N buffers available, and simply work
> on 1 of them at a time in your main application and use the others for
> ongoing non-blocking communication.  This is sometimes called "shadow"
> copies, or "ghost" copies.
>
> Such shadow copies are most useful when you receive something each
> iteration, for example.  For example, something like this:
>
>   buffer[0] = malloc(...);
>   buffer[1] = malloc(...);
>   current = 0;
>   while (still_doing_iterations) {
>   MPI_Irecv(buffer[current], ..., &req);
>   /// work on buffer[current - 1]
>   MPI_Wait(req, MPI_STATUS_IGNORE);
>   current = 1 - current;
>   }
>
> You get the idea.
>
> > 2*) Avoid using temporary message copy because the copy will add extra
> cost on execution time.
>
> It will, if the memcpy cost is significant (especially compared to the
> network time to send it).  If the memcpy is small/insignificant, then don't
> worry about it.
>
> You'll need to determine where this crossover point is, however.
>
> Also keep in mind that MPI and/or the underlying network stack will likely
> be doing these kinds of things under the covers for you.  Indeed, if you
> send short messages -- even via MPI_SEND -- it may return "immediately",
> indicating that MPI says it's safe for you to use the send buffer.  But
> that doesn't mean that the message has even actually left the current
> server and gone out onto the network yet (i.e., some other layer below you
> may have just done a memcpy because it was a short message, and the
> processing/sending of that message is still ongoing).
>
> > And then, we are adviced to do :
> >
> > - replace MPI_SEND by MPI_SSEND (synchroneous blocking sending) : it is
> said that execution is divided by a factor 2
>
> This very, very much depends on your application.
>
> MPI_SSEND won't return until the receiver has started to receive the
> message.
>
> For some communication patterns, putting in this additional level of
> synchronization is helpful -- it keeps all MPI processes in tighter
> synchronization and you might experience less jitter, etc.  And therefore
> overall execution time is faster.
>
> But for others, it adds unnecessary delay.
>
> I'd say it's an over-generalization that simply replacing MPI_SEND with
> MPI_SSEND always reduces execution time by 2.
>
> > - use MPI_ISSEND and MPI_IRECV with MPI_WAIT function to synchronize
> (synchroneous non-blocking sending) : it is said that execution is divided
> by a factor 3
>
> Again, it depends on the app.  Generally, non-blocking communication is
> better -- *if your app can effectively overlap communication and
> computation*.
>
> If your app doesn't take advantage of this overlap, then you won't see
> such performance benefits.  For example:
>
>MPI_Isend(buffer, ..., req);
>MPI_Wait(&req, ...);
>
> Technically, the above uses ISEND and WAIT... but it's actually probably
> going to be *slower* than using MPI_SEND because you've made multiple
> function calls with no additional work between the two -- so the app didn't
> effectively overlap the communication with any local computation.  Hence:
> no performance benefit.
>
> > So what's the best optimization ? Do we have to use temporary message
> copy or not and if yes, what's the case for ?
>
> As you can probably see from my text above, the answer is: it depends.  :-)
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] MacOS - Running MPI code with 256 procs - “[warn] select: Invalid argument” message

[Christophe Petit]

Hello,

Using Open-Mpi 3.0 and following the tutorial on this link
,
I try to run a MPI code under MacOS 10.9.5 (Mavericks) with a number of
process equal to 256 : the MPI code allocates for each process a 512x512 2D
array, so it requires 256*256kB = 64MB of total used memory.

My MacOS has 16GB RAM and 8 cores, so it seems to be weird.

For a number of process lower than 256 (I tried : np=2,4,8,16,32,64,128),
there is no problem, execution is good and I get expected results.

But for np = 256, I get the following message which repeats itself :

$ mpirun -np 256 ./explicitPar

[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
 ...

I tried also to use -mca option by doing :

$ mpirun -mca opal_set_max_sys_limits 1 -np 256 ./explicitPar

But I get the same warning message.

>From this link

cited above, I did :

$ launchctl limit maxfiles

  maxfiles65536  20

Then, in root user, I created /etc/launchd.conf file and put into :

limit maxfiles 65536 20

I restarted the system for the new limits to take effect and type as normal
user :

$ launchctl limit maxfiles

maxfiles65536  20

But unfortunately, these modifications have no effects on the MPI function
"mpirun" with 256 processes and don't make disappear the warning above.

On Linux platform, I can launch my MPI code with np = 256, without problem,
the issue is only happening on MacOS 10.9.5.

I didn't get this issue with previous version of Open-MPI.

Any idea ? Thanks
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MacOS - Running MPI code with 256 procs - “[warn] select: Invalid argument” message

[Christophe Petit]

Thanks for your message.

Actually, if I test ulimit -n, I get :

$ulimit -n
65536

that confirms that modifications (from the tutorial) are taken into account.

To test the OpenMPI "ring_c.c" code, I get :

1) without "--oversubscribe" flag :

$mpirun -np 128 ./ring_c
--
There are not enough slots available in the system to satisfy the 128 slots
that were requested by the application:
  ./a.out

Either request fewer slots for your application, or make more slots
available
for use.
--

2) with "--oversubscribe" flag, everything seems to work fine :

mpirun --oversubscribe -np 128 ./ring_c

Process 0 sending 10 to 1, tag 201 (128 processes in ring)
Process 0 sent to 1
Process 0 decremented value: 9
Process 0 decremented value: 8
Process 0 decremented value: 7
Process 0 decremented value: 6
Process 0 decremented value: 5
Process 0 decremented value: 4
Process 0 decremented value: 3
Process 0 decremented value: 2
Process 0 decremented value: 1
Process 0 decremented value: 0
Process 0 exiting
Process 1 exiting
Process 2 exiting
...

--

Now If I take np = 256 , I get the same warning as my original issue :

$mpirun --oversubscribe -np 256 ./ring_c

[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
...

I have installed Open-MPI 3.0 with the GCC-4.9 from MacPorts
<https://www.macports.org/> ( gcc-mp-4.9) and GFORTRAN also
(gfortran-mp-4.9).

Do you advise me to use gcc CLANG for Open-MPI 3.0 instead or GCC-5 ?

Where can I find a suitable libevent version ? I would like to build
firstly libevent and after Open-MPI 3.0.

Regards


2018-02-15 1:42 GMT+01:00 Gilles Gouaillardet :

> Christophe,
>
> I can only test this on OS X 10.13.3 High Sierra, and it could differ
> from Maverick.
>
> by default
>
> KA15-002:~ gilles$ ulimit -n
> 256
>
> KA15-002:~ gilles$ ulimit -Hn
> unlimited
>
>
> but surprisingly,
>
> KA15-002:~ gilles$ ulimit -n unlimited
> -bash: ulimit: open files: cannot modify limit: Operation not permitted
>
> KA15-002:~ gilles$ ulimit -n 2
> -bash: ulimit: open files: cannot modify limit: Invalid argument
>
> and finally, a lower value works just fine.
>
> KA15-002:~ gilles$ ulimit -n 1
>
>
> as a consequence, opal_set_max_sys_limits fails in my environment.
> --oversubscribe is mandatory (since there are no 256 cores on my
> laptop), and then
>
> ulimit -n 1; mpirun --oversubscribe -np 256 ./ring_c
>
> works just fine (fwiw, this is an example from Open MPI sources
> examples/ring_c.c)
>
>
>
> So first, I invite you to double check with ulimit -n that your system
> changes are effective.
>
> How did you build/install Open MPI ?
> The message seems to come from libevent, and Open MPI uses an embedded
> version of libevent.
> It is possible to use an external version at configure time.
> If you are using an external libevent, you might want to try
> rebuilding Open MPI with the embedded one.
>
>
> Cheers,
>
> Gilles
>
> On Wed, Feb 14, 2018 at 4:57 PM, Christophe Petit
>  wrote:
> > Hello,
> >
> > Using Open-Mpi 3.0 and following the tutorial on this link, I try to run
> a
> > MPI code under MacOS 10.9.5 (Mavericks) with a number of process equal to
> > 256 : the MPI code allocates for each process a 512x512 2D array, so it
> > requires 256*256kB = 64MB of total used memory.
> >
> > My MacOS has 16GB RAM and 8 cores, so it seems to be weird.
> >
> > For a number of process lower than 256 (I tried : np=2,4,8,16,32,64,128),
> > there is no problem, execution is good and I get expected results.
> >
> > But for np = 256, I get the following message which repeats itself :
> >
> > $ mpirun -np 256 ./explicitPar
> >
> > [warn] select: Invalid argument
> > [warn] select: Invalid argument
> > [warn] select: Invalid argument
> >  ...
> >
> > I tried also to use -mca option by doing :
> >
> > $ mpirun -mca opal_set_max_sys_limits 1 -np 256 ./explicitPar
> >
> > But I get the same warning message.
> >
> > From this link cited above, I did :
> >
> > $ launchctl limit maxfiles
> >
> >   maxfiles65536  20
> >
> > Then, in root user, I created

Re: [OMPI users] MacOS - Running MPI code with 256 procs - “[warn] select: Invalid argument” message

[Christophe Petit]

Hello Gilles,

I have upgraded to OS X 10.13.3 High Sierra but still have no luck with my
issue.

To increase the limit of open files, from this link
<https://www.macobserver.com/tips/deep-dive/evade-macos-many-open-files-error-pushing-limits/>
I did :

1) *sudo vim /Library/LaunchDaemons/limit.maxfiles.plist*


2) put into this file :

*
http://www.apple.com/DTDs/PropertyList-1.0.dtd
<http://www.apple.com/DTDs/PropertyList-1.0.dtd>">

 
 Label
 limit.maxfiles
 ProgramArguments
 
 launchctl
 limit
 maxfiles
 64000
 524288
 
 RunAtLoad
 
 ServiceIPC
 
 
*

3) *sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist*

4) Checking by  :



*$ sysctl kern.maxfiles kern.maxfiles: 524288*

and



*$ launchctl limit maxfilesmaxfiles64000  524288 *

When I launch ring_c.c executable with :

*mpirun --oversubscribe -np $NB_PROCESS ./a.out *

For a number of processes lower or equal to *142*, it's running fine

For a number of processes greater than *142*, I get the original warning
message :

[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
...

Why the limit is reached at NB_PROCESS > *142* ? what does it corrrespond
to ?

ps: I have set "*ulimit -n 1*"

Regards, Chris


2018-02-15 1:42 GMT+01:00 Gilles Gouaillardet :

> Christophe,
>
> I can only test this on OS X 10.13.3 High Sierra, and it could differ
> from Maverick.
>
> by default
>
> KA15-002:~ gilles$ ulimit -n
> 256
>
> KA15-002:~ gilles$ ulimit -Hn
> unlimited
>
>
> but surprisingly,
>
> KA15-002:~ gilles$ ulimit -n unlimited
> -bash: ulimit: open files: cannot modify limit: Operation not permitted
>
> KA15-002:~ gilles$ ulimit -n 2
> -bash: ulimit: open files: cannot modify limit: Invalid argument
>
> and finally, a lower value works just fine.
>
> KA15-002:~ gilles$ ulimit -n 1
>
>
> as a consequence, opal_set_max_sys_limits fails in my environment.
> --oversubscribe is mandatory (since there are no 256 cores on my
> laptop), and then
>
> ulimit -n 1; mpirun --oversubscribe -np 256 ./ring_c
>
> works just fine (fwiw, this is an example from Open MPI sources
> examples/ring_c.c)
>
>
>
> So first, I invite you to double check with ulimit -n that your system
> changes are effective.
>
> How did you build/install Open MPI ?
> The message seems to come from libevent, and Open MPI uses an embedded
> version of libevent.
> It is possible to use an external version at configure time.
> If you are using an external libevent, you might want to try
> rebuilding Open MPI with the embedded one.
>
>
> Cheers,
>
> Gilles
>
> On Wed, Feb 14, 2018 at 4:57 PM, Christophe Petit
>  wrote:
> > Hello,
> >
> > Using Open-Mpi 3.0 and following the tutorial on this link, I try to run
> a
> > MPI code under MacOS 10.9.5 (Mavericks) with a number of process equal to
> > 256 : the MPI code allocates for each process a 512x512 2D array, so it
> > requires 256*256kB = 64MB of total used memory.
> >
> > My MacOS has 16GB RAM and 8 cores, so it seems to be weird.
> >
> > For a number of process lower than 256 (I tried : np=2,4,8,16,32,64,128),
> > there is no problem, execution is good and I get expected results.
> >
> > But for np = 256, I get the following message which repeats itself :
> >
> > $ mpirun -np 256 ./explicitPar
> >
> > [warn] select: Invalid argument
> > [warn] select: Invalid argument
> > [warn] select: Invalid argument
> >  ...
> >
> > I tried also to use -mca option by doing :
> >
> > $ mpirun -mca opal_set_max_sys_limits 1 -np 256 ./explicitPar
> >
> > But I get the same warning message.
> >
> > From this link cited above, I did :
> >
> > $ launchctl limit maxfiles
> >
> >   maxfiles65536  20
> >
> > Then, in root user, I created /etc/launchd.conf file and put into :
> >
> > limit maxfiles 65536 20
> >
> > I restarted the system for the new limits to take effect and type as
> normal
> > user :
> >
> > $ launchctl limit maxfiles
> >
> > maxfiles65536  20
> >
> > But unfortunately, these modifications have no effects on the MPI
> function
> > "mpirun" with 256 processes and don't make disappear the warning above.
> >
> > On Linux platform, I can launch my MPI code with np = 256, without
> problem,
> > the issue is only happening on MacOS 10.9.5.
> >
> > I didn't get this issue with previous version of Open-MPI.
> >
> > Any idea ? Thanks
> >
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MacOS - Running MPI code with 256 procs - “[warn] select: Invalid argument” message

[Christophe Petit]

Hello Gilles,

I have finally fixed my problem by using external libevent for open-mpi-3.0
(built from libevent-2.1.8 version) and the following configure :

 ./configure --prefix=/usr/local/openmpi CC=gcc-mp-7 CXX=g++-mp-7
FC=gfortran-mp-7 --with-libevent=external
--with-libevent=/usr/local/libevent/

you were right when indicating me to look for a solution with libevent.

Thanks, Chris



2018-02-20 6:39 GMT+01:00 Gilles Gouaillardet :

> Christophe,
>
>
> could you try the hello_c.c and ring_c.c from the examples directory ?
>
> If it still does not work, then could you give a try to Open MPI from
> homebrew ?
>
> and if that fixes the issue, can you please post your configure command
> line ?
>
>
> Cheers,
>
>
> Gilles
>
>
> On 2/17/2018 7:08 AM, Christophe Petit wrote:
>
>> Hello Gilles,
>>
>> I have upgraded to OS X 10.13.3 High Sierra but still have no luck with
>> my issue.
>>
>> To increase the limit of open files, from this link <
>> https://www.macobserver.com/tips/deep-dive/evade-macos-many
>> -open-files-error-pushing-limits/> I did :
>>
>> 1)*sudo vim /Library/LaunchDaemons/limit.maxfiles.plist*
>>
>> 2) put into this file :
>> * > "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/Pro
>> pertyList-1.0.dtd">   Label
>> limit.maxfiles ProgramArguments 
>> launchctl limit maxfiles
>> 64000 524288 
>> RunAtLoad  ServiceIPC  
>> *
>> 3) *sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist*
>>
>> 4) Checking by  :
>>
>> *$ sysctl kern.maxfiles
>>
>> kern.maxfiles: 524288*
>>
>> and
>>
>> *$ launchctl limit maxfiles
>>
>> maxfiles64000  524288 *
>>
>> When I launch ring_c.c executable with :
>>
>> *mpirun --oversubscribe -np $NB_PROCESS ./a.out *
>>
>> For a number of processes lower or equal to *142*, it's running fine
>>
>> For a number of processes greater than *142*, I get the original warning
>> message :
>>
>> [warn] select: Invalid argument
>> [warn] select: Invalid argument
>> [warn] select: Invalid argument
>> [warn] select: Invalid argument
>> [warn] select: Invalid argument
>> [warn] select: Invalid argument
>> [warn] select: Invalid argument
>> ...
>>
>> Why the limit is reached at NB_PROCESS > *142* ? what does it corrrespond
>> to ?
>>
>> ps: I have set "*ulimit -n 1*"
>>
>> Regards, Chris
>>
>>
>> 2018-02-15 1:42 GMT+01:00 Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>>:
>>
>>
>> Christophe,
>>
>> I can only test this on OS X 10.13.3 High Sierra, and it could differ
>> from Maverick.
>>
>> by default
>>
>> KA15-002:~ gilles$ ulimit -n
>> 256
>>
>> KA15-002:~ gilles$ ulimit -Hn
>> unlimited
>>
>>
>> but surprisingly,
>>
>> KA15-002:~ gilles$ ulimit -n unlimited
>> -bash: ulimit: open files: cannot modify limit: Operation not
>> permitted
>>
>> KA15-002:~ gilles$ ulimit -n 2
>> -bash: ulimit: open files: cannot modify limit: Invalid argument
>>
>> and finally, a lower value works just fine.
>>
>> KA15-002:~ gilles$ ulimit -n 1
>>
>>
>> as a consequence, opal_set_max_sys_limits fails in my environment.
>> --oversubscribe is mandatory (since there are no 256 cores on my
>> laptop), and then
>>
>> ulimit -n 10000; mpirun --oversubscribe -np 256 ./ring_c
>>
>> works just fine (fwiw, this is an example from Open MPI sources
>> examples/ring_c.c)
>>
>>
>>
>> So first, I invite you to double check with ulimit -n that your system
>> changes are effective.
>>
>> How did you build/install Open MPI ?
>> The message seems to come from libevent, and Open MPI uses an embedded
>> version of libevent.
>> It is possible to use an external version at configure time.
>> If you are using an external libevent, you might want to try
>> rebuilding Open MPI with the embedded one.
>>
>>
>> Cheers,
>>
>> Gilles
>>
>> On Wed, Feb 14, 2018 at 4:57 PM, Christophe Petit
>> > <mailto:christophe.peti...@gmail.com>> wrote:
>> > Hello,
>> >
>> > Using Open-Mpi 3.0 and following the tutorial on th