[OMPI users] (no subject)
Hi
I am trying to run the following simple demo to a cluster of two nodes
--
#include
#include
int main(int argc, char** argv) {
MPI_Init(NULL, NULL);
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
printf("Hello world from processor %s, rank %d" " out of %d
processors\n", processor_name, world_rank, world_size);
MPI_Finalize();
}
-
i get always the message
It looks like opal_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during opal_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
opal_shmem_base_select failed
--> Returned value -1 instead of OPAL_SUCCESS
--
any hint?
Ioannis Botsis
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] (no subject)
Ioannis,
### What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git
branch name and hash, etc.)
### Describe how Open MPI was installed (e.g., from a source/
distribution tarball, from a git clone, from an operating system
distribution package, etc.)
### Please describe the system on which you are running
* Operating system/version:
* Computer hardware:
* Network type:
also, what if you
mpirun --mca shmem_base_verbose 100 ...
Cheers,
Gilles
- Original Message -
> Hi
>
> I am trying to run the following simple demo to a cluster of two nodes
>
> --
> #include
> #include
>
> int main(int argc, char** argv) {
> MPI_Init(NULL, NULL);
>
> int world_size;
> MPI_Comm_size(MPI_COMM_WORLD, &world_size);
>
> int world_rank;
> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
>
> char processor_name[MPI_MAX_PROCESSOR_NAME];
> int name_len;
> MPI_Get_processor_name(processor_name, &name_len);
>
> printf("Hello world from processor %s, rank %d" " out of %d
> processors\n", processor_name, world_rank, world_size);
>
> MPI_Finalize();
> }
> --
---
>
> i get always the message
>
> --
--
> It looks like opal_init failed for some reason; your parallel process
is
> likely to abort. There are many reasons that a parallel process can
> fail during opal_init; some of which are due to configuration or
> environment problems. This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
>opal_shmem_base_select failed
>--> Returned value -1 instead of OPAL_SUCCESS
> --
>
> any hint?
>
> Ioannis Botsis
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
[OMPI users] Form an intercommunicator across threads?
I am trying to craft a client-server layer that needs to have 2 different modes of operation. In the “remote server” mode, then the server runs on distinct processes, and intercommunicator is a perfect fit for my design. In the “local server” the server will actually run on a dedicate thread within the same pool of processes that are running the client. What I would like is some analog of an intercommunicator that would operate between two comms that are essentially identical (but distinct due to use of MPI_Comm_dup()), with one running on thread 0 and the other running on thread 1. Now, I know the language in the standard sounds like it explicitly forbids such overlapping groups: "Overlap of local and remote groups that are bound into an inter-communicator is prohibited.” But there is then that hint of wiggle room with the parenthetical: “(If a process is multithreaded, and MPI calls block only a thread, rather than a process, then “dual membership” can be supported. It is then the user’s responsibility to make sure that calls on behalf of the two “roles” of a process are executed by two independent threads.)" This sounds very much like what I want to do, but the following example code crashes (apologies for Fortran source).It is intended to be run on two threads - each of which creates a dup of MPI_COMM_WORLD. The intercomm create would then have the “other” comm as the remote for each thread. I’m not wed to the details here. Anything that would support the notion of an MPI based server running on extra threads would be a potential solution. I know that in the worst case, I could achieve the same by oversubscribing processes on my nodes, but that pushes some issues in our batch processing system that I’d prefer to avoid. A definitive “Can’t be done. Move along.” would also be useful. The following fails with OpenMPI 2.1 on OS X 10.12: program main use mpi implicit none integer, external :: omp_get_thread_num integer :: ierror integer :: comms(0:1) integer :: comm integer :: tag = 10 integer :: intercomm integer :: provided integer :: i call MPI_Init_thread(MPI_THREAD_MULTIPLE, provided, ierror) call MPI_Comm_dup(MPI_COMM_WORLD, comm, ierror) !$omp parallel num_threads(2) default(none), private(i, intercomm, ierror), shared(tag, comms, comm) i = omp_get_thread_num() if (i == 0) then call MPI_Comm_dup(MPI_COMM_WORLD, comms(i), ierror) end if !$omp barrier if (i == 1) then call MPI_Comm_dup(MPI_COMM_WORLD, comms(i), ierror) end if !$omp barrier call MPI_Intercomm_create(comms(i), 0, comm, 0, tag, intercomm, ierror) !$omp end parallel stop call MPI_Finalize(ierror) end program main Thanks in advance. - Tom ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] mpi_scatterv problem in fortran
Based upon the symbols in the backtrace, you are using Intel MPI, not Open-MPI. If there is a bug in the MPI library, it is likely also in MPICH, so you might try to reproduce this in MPICH. You can also try to run with Open-MPI. If you see a problem in both Intel MPI/MPICH and Open-MPI, it is almost certainly incorrect usage. Jeff On Sun, May 14, 2017 at 11:30 PM, Siva Srinivas Kolukula < allwayzit...@gmail.com> wrote: > I want to scatter matrix from root to other processors using scatterv. I > am creating a communicator topology using *mpi_cart_create*. As an > example I have the below code in fortran: > > PROGRAM SendRecv > USE mpi > IMPLICIT none > integer, PARAMETER :: m = 4, n = 4 > integer, DIMENSION(m,n) :: a, b,h > integer :: i,j,count > integer,allocatable, dimension(:,:):: loc ! local piece of global 2d array > INTEGER :: istatus(MPI_STATUS_SIZE),ierr > integer, dimension(2) :: sizes, subsizes, starts > INTEGER :: ista,iend,jsta,jend,ilen,jlen > INTEGER :: iprocs, jprocs, nprocs > integer,allocatable,dimension(:):: rcounts, displs > INTEGER :: rcounts0,displs0 > integer, PARAMETER :: ROOT = 0 > integer :: dims(2),coords(2) > logical :: periods(2) > data periods/2*.false./ > integer :: status(MPI_STATUS_SIZE) > integer :: comm2d,source,myrank > integer :: newtype, resizedtype > integer :: comsize,charsize > integer(kind=MPI_ADDRESS_KIND) :: extent, begin > > CALL MPI_INIT(ierr) > CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nprocs, ierr) > CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank, ierr) > ! Get a new communicator for a decomposition of the domain. > dims(1) = 0 > dims(2) = 0 > CALL MPI_DIMS_CREATE(nprocs,2,dims,ierr) > if (myrank.EQ.Root) then >print *,nprocs,'processors have been arranged > into',dims(1),'X',dims(2),'grid' > endif > CALL MPI_CART_CREATE(MPI_COMM_WORLD,2,dims,periods,.true., & > comm2d,ierr) > ! Get my position in this communicator > CALL MPI_COMM_RANK(comm2d,myrank,ierr) > ! Get the decomposition > CALL fnd2ddecomp(comm2d,m,n,ista,iend,jsta,jend) > ! print *,ista,jsta,iend,jend > ilen = iend - ista + 1 > jlen = jend - jsta + 1 > > CALL MPI_Cart_get(comm2d,2,dims,periods,coords,ierr) > iprocs = dims(1) > jprocs = dims(2) > ! define the global matrix > if (myrank==ROOT) then >count = 0 > do j = 1,n >do i = 1,m > a(i,j) = count > count = count+1 >enddo > enddo > print *, 'global matrix is: ' > do 90 i=1,m >do 80 j = 1,n >write(*,70)a(i,j) > 70 format(2x,I5,$) > 80 continue >print *, ' ' > 90continue > endif > call MPI_Barrier(MPI_COMM_WORLD, ierr) > > starts = [0,0] > sizes= [m, n] > subsizes = [ilen, jlen] > call MPI_Type_create_subarray(2, sizes, subsizes, starts,& >MPI_ORDER_FORTRAN, MPI_INTEGER, & >newtype, ierr) > call MPI_Type_size(MPI_INTEGER, charsize, ierr) > begin = 0 > extent = charsize > call MPI_Type_create_resized(newtype, begin, extent, resizedtype, ierr) > call MPI_Type_commit(resizedtype, ierr) > > ! get counts and displacmeents > allocate(rcounts(nprocs),displs(nprocs)) > rcounts0 = 1 > displs0 = (ista-1) + (jsta-1)*m > CALL MPI_Allgather(rcounts0,1,MPI_INT,rcounts,1,MPI_INT,MPI_COMM_WORLD,IERR) > CALL MPI_Allgather(displs0,1,MPI_INT,displs,1,MPI_INT,MPI_COMM_WORLD,IERR) > CALL MPI_Barrier(MPI_COMM_WORLD, ierr) > > ! scatter data > allocate(loc(ilen,jlen)) > call MPI_Scatterv(a,rcounts,displs,resizedtype,& > loc,ilen*jlen,MPI_INTEGER, & > ROOT,MPI_COMM_WORLD,ierr) > ! print each processor matrix > do source = 0,nprocs-1 >if (myrank.eq.source) then >print *,'myrank:',source >do i=1,ilen >do j = 1,jlen > write(*,701)loc(i,j) > 701 format(2x,I5,$) >enddo >print *, ' ' >enddo > endif >call MPI_Barrier(MPI_COMM_WORLD, ierr) > enddo > > call MPI_Type_free(newtype,ierr) > call MPI_Type_free(resizedtype,ierr) > deallocate(rcounts,displs) > deallocate(loc) > > CALL MPI_FINALIZE(ierr) > > contains > > subroutine fnd2ddecomp(comm2d,m,n,ista,iend,jsta,jend) > integer comm2d > integer m,n,ista,jsta,iend,jend > integer dims(2),coords(2),ierr > logical periods(2) > ! Get (i,j) position of a processor from Cartesian topology. > CALL MPI_Cart_get(comm2d,2,dims,periods,coords,ierr) > ! Decomposition in first (ie. X) direction > CALL MPE_DECOMP1D(m,dims(1),coords(1),ista,iend) > ! Decomposition in second (ie. Y) direction > CALL MPE_DECOMP1D(n,dims(2),coords(2),jsta,jend) > end subroutine fnd2ddecomp > > SUBROUTINE MPE_DECOMP1D(n,numprocs,myid,s,e) > integer n,numprocs,myid,s,e,nlocal,deficit > nlocal = n / numprocs > s = myid * nlocal + 1 > deficit = mod(n,numprocs) > s = s + min(myid,deficit) > ! Give one more slice to processors > if (myid .lt. deficit) then > nlocal = nlocal + 1 > endif > e = s + nlocal - 1
Re: [OMPI users] Form an intercommunicator across threads?
A process or rank is not allowed to participate multiple times in the same group (at least not in the current version of the MPI standard). The sentence about "dual membership" you pointed out makes sense only for inter-communicators (and the paragraph where the sentence is located clearly talks about local and remote groups). Moreover, and this is something the MPI standard is not clear about, such a sentence only makes sense when the local_leader of the 2 groups is not the same process. The problem with your approach is that you have in the peer_comm the same process as being the leader of the 2 groups. In the implementation of the MPI_Intercomm_create there is a need for a handshake between the 2 leaders to exchange the sizes of their respective groups, and this handshake is using the TAG you provided. The problem (at least in Open MPI) is that the handshake is implemented as (Irecv + Send + Wait) with the same rank on the peer_comm. Obviously such a communication pattern will not successfully exchange the data between the peers in most cases (in the same communicator the matching is guaranteed to be in FIFO order by the MPI standard). So the complete answer is: It can be done but not in the context you are trying to achieve. George. On Mon, May 15, 2017 at 10:22 AM, Clune, Thomas L. (GSFC-6101) < thomas.l.cl...@nasa.gov> wrote: > I am trying to craft a client-server layer that needs to have 2 different > modes of operation. In the “remote server” mode, then the server runs on > distinct processes, and intercommunicator is a perfect fit for my design. > In the “local server” the server will actually run on a dedicate thread > within the same pool of processes that are running the client. What I > would like is some analog of an intercommunicator that would operate > between two comms that are essentially identical (but distinct due to use > of MPI_Comm_dup()), with one running on thread 0 and the other running on > thread 1. > > Now, I know the language in the standard sounds like it explicitly forbids > such overlapping groups: > > "Overlap of local and remote groups that are bound into an > inter-communicator is prohibited.” > > But there is then that hint of wiggle room with the parenthetical: > > “(If a process is multithreaded, and MPI calls block only a thread, > rather than a process, then “dual membership” can be supported. It is > then the user’s responsibility to make sure that calls on behalf of the two > “roles” of a process are executed by two independent threads.)" > > This sounds very much like what I want to do, but the following example > code crashes (apologies for Fortran source).It is intended to be run on > two threads - each of which creates a dup of MPI_COMM_WORLD. The > intercomm create would then have the “other” comm as the remote for each > thread. I’m not wed to the details here. Anything that would > support the notion of an MPI based server running on extra threads would be > a potential solution. I know that in the worst case, I could achieve the > same by oversubscribing processes on my nodes, but that pushes some issues > in our batch processing system that I’d prefer to avoid. A definitive “ > Can’t be done. Move along.” would also be useful. > > The following fails with OpenMPI 2.1 on OS X 10.12: > > program main >use mpi >implicit none > >integer, external :: omp_get_thread_num >integer :: ierror >integer :: comms(0:1) >integer :: comm >integer :: tag = 10 >integer :: intercomm >integer :: provided >integer :: i > >call MPI_Init_thread(MPI_THREAD_MULTIPLE, provided, ierror) >call MPI_Comm_dup(MPI_COMM_WORLD, comm, ierror) > >!$omp parallel num_threads(2) default(none), private(i, intercomm, > ierror), shared(tag, comms, comm) > i = omp_get_thread_num() > > if (i == 0) then > call MPI_Comm_dup(MPI_COMM_WORLD, comms(i), ierror) > end if > !$omp barrier > if (i == 1) then > call MPI_Comm_dup(MPI_COMM_WORLD, comms(i), ierror) > end if > !$omp barrier > call MPI_Intercomm_create(comms(i), 0, comm, 0, tag, intercomm, > ierror) > >!$omp end parallel > >stop >call MPI_Finalize(ierror) > end program main > > > Thanks in advance. > > - Tom > > > > ___ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > ___ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
Re: [OMPI users] (no subject)
Hi Gilles Thank you for your prompt response. Here is some information about the system Ubuntu 16.04 server Linux-4.4.0-75-generic-x86_64-with-Ubuntu-16.04-xenial On HP PROLIANT DL320R05 Generation 5, 4GB RAM, 4x120GB raid-1 HDD, 2 ethernet ports 10/100/1000 HP StorageWorks 70 Modular Smart Array with 14x120GB HDD (RAID-5) 44 HP Proliant BL465c server blade, double AMD Opteron Model 2218(2.6GHz, 2MB, 95W), 4 GB RAM, 2 NC370i Multifunction Gigabit Servers Adapters, 120GB User's area is shared with the nodes. ssh and torque 6.0.2 services works fine Torque and openmpi 2.1.0 are installed from tarball. configure --prefix=/storage/exp_soft/tuc is used for the deployment of openmpi 2.1.0. After make and make install binaries, lib and include files of openmpi2.1.0 are located under /storage/exp_soft/tuc . /storage is a shared file system for all the nodes of the cluster $PATH: /storage/exp_soft/tuc/bin /storage/exp_soft/tuc/sbin /storage/exp_soft/tuc/torque/bin /storage/exp_soft/tuc/torque/sbin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /snap/bin LD_LIBRARY_PATH=/storage/exp_soft/tuc/lib C_INCLUDE_PATH=/storage/exp_soft/tuc/include I use also jupyterhub (with cluster tab enabled) as a user interface to the cluster. After the installation of python and some dependencies mpich and openmpi are also installed in the system directories. -- mpirun --allow-run-as-root --mca shmem_base_verbose 100 ... [se01.grid.tuc.gr:19607] mca: base: components_register: registering framework shmem components [se01.grid.tuc.gr:19607] mca: base: components_register: found loaded component sysv [se01.grid.tuc.gr:19607] mca: base: components_register: component sysv register function successful [se01.grid.tuc.gr:19607] mca: base: components_register: found loaded component posix [se01.grid.tuc.gr:19607] mca: base: components_register: component posix register function successful [se01.grid.tuc.gr:19607] mca: base: components_register: found loaded component mmap [se01.grid.tuc.gr:19607] mca: base: components_register: component mmap register function successful [se01.grid.tuc.gr:19607] mca: base: components_open: opening shmem components [se01.grid.tuc.gr:19607] mca: base: components_open: found loaded component sysv [se01.grid.tuc.gr:19607] mca: base: components_open: component sysv open function successful [se01.grid.tuc.gr:19607] mca: base: components_open: found loaded component posix [se01.grid.tuc.gr:19607] mca: base: components_open: component posix open function successful [se01.grid.tuc.gr:19607] mca: base: components_open: found loaded component mmap [se01.grid.tuc.gr:19607] mca: base: components_open: component mmap open function successful [se01.grid.tuc.gr:19607] shmem: base: runtime_query: Auto-selecting shmem components [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Querying component (run-time) [sysv] [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Query of component [sysv] set priority to 30 [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Querying component (run-time) [posix] [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Query of component [posix] set priority to 40 [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Querying component (run-time) [mmap] [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Query of component [mmap] set priority to 50 [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Selected component [mmap] [se01.grid.tuc.gr:19607] mca: base: close: unloading component sysv [se01.grid.tuc.gr:19607] mca: base: close: unloading component posix [se01.grid.tuc.gr:19607] shmem: base: best_runnable_component_name: Searching for best runnable component. [se01.grid.tuc.gr:19607] shmem: base: best_runnable_component_name: Found best runnable component: (mmap). -- mpirun was unable to find the specified executable file, and therefore did not launch the job. This error was first reported for process rank 0; it may have occurred for other processes as well. NOTE: A common cause for this error is misspelling a mpirun command line parameter option (remember that mpirun interprets the first unrecognized command line token as the executable). Node: se01 Executable: ... -- 2 total processes failed to start [se01.grid.tuc.gr:19607] mca: base: close: component mmap closed [se01.grid.tuc.gr:19607] mca: base: close: unloading component mmap jb -Original Message- From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of gil...@rist.or.jp Sent: Monday, May
Re: [OMPI users] mpi_scatterv problem in fortran
Dear Jeff Hammond Thanks a lot for the reply. I have tried with mpiexec, I am getting the same error. But according to this link: http://stackoverflow.com/questions/7549316/mpi-partition-matrix-into-blocks it is possible. Any suggestions/ advice? _ *SAVE WATER ** ~ **SAVE ENERGY**~ **~ **SAVE EARTH *[image: Earth-22-june.gif (7996 bytes)] http://sites.google.com/site/kolukulasivasrinivas/ Siva Srinivas Kolukula, PhD *Scientist - B* Indian Tsunami Early Warning Centre (ITEWC) Advisory Services and Satellite Oceanography Group (ASG) Indian National Centre for Ocean Information Services (INCOIS) "Ocean Valley" Pragathi Nagar (B.O) Nizampet (S.O) Hyderabad - 500 090 Telangana, INDIA Office: 040 23886124 *Cell: +91 9381403232; +91 8977801947* On Mon, May 15, 2017 at 8:32 PM, Jeff Hammond wrote: > Based upon the symbols in the backtrace, you are using Intel MPI, not > Open-MPI. If there is a bug in the MPI library, it is likely also in > MPICH, so you might try to reproduce this in MPICH. You can also try to > run with Open-MPI. If you see a problem in both Intel MPI/MPICH and > Open-MPI, it is almost certainly incorrect usage. > > Jeff > > On Sun, May 14, 2017 at 11:30 PM, Siva Srinivas Kolukula < > allwayzit...@gmail.com> wrote: > >> I want to scatter matrix from root to other processors using scatterv. I >> am creating a communicator topology using *mpi_cart_create*. As an >> example I have the below code in fortran: >> >> PROGRAM SendRecv >> USE mpi >> IMPLICIT none >> integer, PARAMETER :: m = 4, n = 4 >> integer, DIMENSION(m,n) :: a, b,h >> integer :: i,j,count >> integer,allocatable, dimension(:,:):: loc ! local piece of global 2d array >> INTEGER :: istatus(MPI_STATUS_SIZE),ierr >> integer, dimension(2) :: sizes, subsizes, starts >> INTEGER :: ista,iend,jsta,jend,ilen,jlen >> INTEGER :: iprocs, jprocs, nprocs >> integer,allocatable,dimension(:):: rcounts, displs >> INTEGER :: rcounts0,displs0 >> integer, PARAMETER :: ROOT = 0 >> integer :: dims(2),coords(2) >> logical :: periods(2) >> data periods/2*.false./ >> integer :: status(MPI_STATUS_SIZE) >> integer :: comm2d,source,myrank >> integer :: newtype, resizedtype >> integer :: comsize,charsize >> integer(kind=MPI_ADDRESS_KIND) :: extent, begin >> >> CALL MPI_INIT(ierr) >> CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nprocs, ierr) >> CALL MPI_COMM_RANK(MPI_COMM_WORLD, myrank, ierr) >> ! Get a new communicator for a decomposition of the domain. >> dims(1) = 0 >> dims(2) = 0 >> CALL MPI_DIMS_CREATE(nprocs,2,dims,ierr) >> if (myrank.EQ.Root) then >>print *,nprocs,'processors have been arranged >> into',dims(1),'X',dims(2),'grid' >> endif >> CALL MPI_CART_CREATE(MPI_COMM_WORLD,2,dims,periods,.true., & >> comm2d,ierr) >> ! Get my position in this communicator >> CALL MPI_COMM_RANK(comm2d,myrank,ierr) >> ! Get the decomposition >> CALL fnd2ddecomp(comm2d,m,n,ista,iend,jsta,jend) >> ! print *,ista,jsta,iend,jend >> ilen = iend - ista + 1 >> jlen = jend - jsta + 1 >> >> CALL MPI_Cart_get(comm2d,2,dims,periods,coords,ierr) >> iprocs = dims(1) >> jprocs = dims(2) >> ! define the global matrix >> if (myrank==ROOT) then >>count = 0 >> do j = 1,n >>do i = 1,m >> a(i,j) = count >> count = count+1 >>enddo >> enddo >> print *, 'global matrix is: ' >> do 90 i=1,m >>do 80 j = 1,n >>write(*,70)a(i,j) >> 70 format(2x,I5,$) >> 80 continue >>print *, ' ' >> 90continue >> endif >> call MPI_Barrier(MPI_COMM_WORLD, ierr) >> >> starts = [0,0] >> sizes= [m, n] >> subsizes = [ilen, jlen] >> call MPI_Type_create_subarray(2, sizes, subsizes, starts,& >>MPI_ORDER_FORTRAN, MPI_INTEGER, & >>newtype, ierr) >> call MPI_Type_size(MPI_INTEGER, charsize, ierr) >> begin = 0 >> extent = charsize >> call MPI_Type_create_resized(newtype, begin, extent, resizedtype, ierr) >> call MPI_Type_commit(resizedtype, ierr) >> >> ! get counts and displacmeents >> allocate(rcounts(nprocs),displs(nprocs)) >> rcounts0 = 1 >> displs0 = (ista-1) + (jsta-1)*m >> CALL MPI_Allgather(rcounts0,1,MPI_INT,rcounts,1,MPI_INT,MPI_COMM_WORLD,IERR) >> CALL MPI_Allgather(displs0,1,MPI_INT,displs,1,MPI_INT,MPI_COMM_WORLD,IERR) >> CALL MPI_Barrier(MPI_COMM_WORLD, ierr) >> >> ! scatter data >> allocate(loc(ilen,jlen)) >> call MPI_Scatterv(a,rcounts,displs,resizedtype,& >> loc,ilen*jlen,MPI_INTEGER, & >> ROOT,MPI_COMM_WORLD,ierr) >> ! print each processor matrix >> do source = 0,nprocs-1 >>if (myrank.eq.source) then >>print *,'myrank:',source >>do i=1,ilen >>do j = 1,jlen >> write(*,701)loc(i,j) >> 701 format(2x,I5,$) >>enddo >>print *, ' ' >>enddo >> endif >>call MPI_Barrier(MPI_COMM_WORLD, ierr) >> enddo >> >> call MPI_Type_free(newtype,ierr) >> call MPI_Type_free(resiz
Re: [OMPI users] mpi_scatterv problem in fortran
Hi,
if you run this under a debugger and look at how MPI_Scatterv is
invoked, you will find that
- sendcounts = {1, 1, 1}
- resizedtype has size 32
- recvcount*sizeof(MPI_INTEGER) = 32 on task 0, but 16 on task 1 and 2
=> too much data is sent to tasks 1 and 2, hence the error.
in this case (3 MPI tasks), my best bet is
- sendcounts should be {2, 1, 1}
- resizedtype should be a row (e.g. 16 bytes)
- recvcount is fine (e.g. 8,4,4)
since the program fails on both MPICH and OpenMPI, the error is most
likely in the program itself
Best regards,
Gilles
On 5/15/2017 3:30 PM, Siva Srinivas Kolukula wrote:
I want to scatter matrix from root to other processors using scatterv.
I am creating a communicator topology using /mpi_cart_create/. As an
example I have the below code in fortran:
|PROGRAM SendRecv USE mpi IMPLICIT none integer, PARAMETER :: m = 4, n
= 4 integer, DIMENSION(m,n) :: a, b,h integer :: i,j,count
integer,allocatable, dimension(:,:):: loc ! local piece of global 2d
array INTEGER :: istatus(MPI_STATUS_SIZE),ierr integer, dimension(2)
:: sizes, subsizes, starts INTEGER :: ista,iend,jsta,jend,ilen,jlen
INTEGER :: iprocs, jprocs, nprocs integer,allocatable,dimension(:)::
rcounts, displs INTEGER :: rcounts0,displs0 integer, PARAMETER :: ROOT
= 0 integer :: dims(2),coords(2) logical :: periods(2) data
periods/2*.false./ integer :: status(MPI_STATUS_SIZE) integer ::
comm2d,source,myrank integer :: newtype, resizedtype integer ::
comsize,charsize integer(kind=MPI_ADDRESS_KIND) :: extent, begin CALL
MPI_INIT(ierr) CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nprocs, ierr) CALL
MPI_COMM_RANK(MPI_COMM_WORLD, myrank, ierr) ! Get a new communicator
for a decomposition of the domain. dims(1) = 0 dims(2) = 0 CALL
MPI_DIMS_CREATE(nprocs,2,dims,ierr) if (myrank.EQ.Root) then print
*,nprocs,'processors have been arranged
into',dims(1),'X',dims(2),'grid' endif CALL
MPI_CART_CREATE(MPI_COMM_WORLD,2,dims,periods,.true., & comm2d,ierr) !
Get my position in this communicator CALL
MPI_COMM_RANK(comm2d,myrank,ierr) ! Get the decomposition CALL
fnd2ddecomp(comm2d,m,n,ista,iend,jsta,jend) ! print
*,ista,jsta,iend,jend ilen = iend - ista + 1 jlen = jend - jsta + 1
CALL MPI_Cart_get(comm2d,2,dims,periods,coords,ierr) iprocs = dims(1)
jprocs = dims(2) ! define the global matrix if (myrank==ROOT) then
count = 0 do j = 1,n do i = 1,m a(i,j) = count count = count+1 enddo
enddo print *, 'global matrix is: ' do 90 i=1,m do 80 j = 1,n
write(*,70)a(i,j) 70 format(2x,I5,$) 80 continue print *, ' ' 90
continue endif call MPI_Barrier(MPI_COMM_WORLD, ierr) starts = [0,0]
sizes = [m, n] subsizes = [ilen, jlen] call
MPI_Type_create_subarray(2, sizes, subsizes, starts, &
MPI_ORDER_FORTRAN, MPI_INTEGER, & newtype, ierr) call
MPI_Type_size(MPI_INTEGER, charsize, ierr) begin = 0 extent = charsize
call MPI_Type_create_resized(newtype, begin, extent, resizedtype,
ierr) call MPI_Type_commit(resizedtype, ierr) ! get counts and
displacmeents allocate(rcounts(nprocs),displs(nprocs)) rcounts0 = 1
displs0 = (ista-1) + (jsta-1)*m CALL
MPI_Allgather(rcounts0,1,MPI_INT,rcounts,1,MPI_INT,MPI_COMM_WORLD,IERR)
CALL
MPI_Allgather(displs0,1,MPI_INT,displs,1,MPI_INT,MPI_COMM_WORLD,IERR)
CALL MPI_Barrier(MPI_COMM_WORLD, ierr) ! scatter data
allocate(loc(ilen,jlen)) call
MPI_Scatterv(a,rcounts,displs,resizedtype, &
loc,ilen*jlen,MPI_INTEGER, & ROOT,MPI_COMM_WORLD,ierr) ! print each
processor matrix do source = 0,nprocs-1 if (myrank.eq.source) then
print *,'myrank:',source do i=1,ilen do j = 1,jlen
write(*,701)loc(i,j) 701 format(2x,I5,$) enddo print *, ' ' enddo
endif call MPI_Barrier(MPI_COMM_WORLD, ierr) enddo call
MPI_Type_free(newtype,ierr) call MPI_Type_free(resizedtype,ierr)
deallocate(rcounts,displs) deallocate(loc) CALL MPI_FINALIZE(ierr)
contains subroutine fnd2ddecomp(comm2d,m,n,ista,iend,jsta,jend)
integer comm2d integer m,n,ista,jsta,iend,jend integer
dims(2),coords(2),ierr logical periods(2) ! Get (i,j) position of a
processor from Cartesian topology. CALL
MPI_Cart_get(comm2d,2,dims,periods,coords,ierr) ! Decomposition in
first (ie. X) direction CALL
MPE_DECOMP1D(m,dims(1),coords(1),ista,iend) ! Decomposition in second
(ie. Y) direction CALL MPE_DECOMP1D(n,dims(2),coords(2),jsta,jend) end
subroutine fnd2ddecomp SUBROUTINE MPE_DECOMP1D(n,numprocs,myid,s,e)
integer n,numprocs,myid,s,e,nlocal,deficit nlocal = n / numprocs s =
myid * nlocal + 1 deficit = mod(n,numprocs) s = s + min(myid,deficit)
! Give one more slice to processors if (myid .lt. deficit) then nlocal
= nlocal + 1 endif e = s + nlocal - 1 if (e .gt. n .or. myid .eq.
numprocs-1) e = n end subroutine MPE_DECOMP1D END program SendRecv |
I am generating a 4x4 matrix, and using scatterv I am sending the
blocks of matrices to other processors. Code works fine for 4,2 and 16
processors. But throws a error for three processors. What
modifications I have to do make it work for any number of given
processors.
Global matri
Re: [OMPI users] (no subject)
Thanks for all the information, what i meant by mpirun --mca shmem_base_verbose 100 ... is really you modify your mpirun command line (or your torque script if applicable) and add --mca shmem_base_verbose 100 right after mpirun Cheers, Gilles On 5/16/2017 3:59 AM, Ioannis Botsis wrote: Hi Gilles Thank you for your prompt response. Here is some information about the system Ubuntu 16.04 server Linux-4.4.0-75-generic-x86_64-with-Ubuntu-16.04-xenial On HP PROLIANT DL320R05 Generation 5, 4GB RAM, 4x120GB raid-1 HDD, 2 ethernet ports 10/100/1000 HP StorageWorks 70 Modular Smart Array with 14x120GB HDD (RAID-5) 44 HP Proliant BL465c server blade, double AMD Opteron Model 2218(2.6GHz, 2MB, 95W), 4 GB RAM, 2 NC370i Multifunction Gigabit Servers Adapters, 120GB User's area is shared with the nodes. ssh and torque 6.0.2 services works fine Torque and openmpi 2.1.0 are installed from tarball. configure --prefix=/storage/exp_soft/tuc is used for the deployment of openmpi 2.1.0. After make and make install binaries, lib and include files of openmpi2.1.0 are located under /storage/exp_soft/tuc . /storage is a shared file system for all the nodes of the cluster $PATH: /storage/exp_soft/tuc/bin /storage/exp_soft/tuc/sbin /storage/exp_soft/tuc/torque/bin /storage/exp_soft/tuc/torque/sbin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /snap/bin LD_LIBRARY_PATH=/storage/exp_soft/tuc/lib C_INCLUDE_PATH=/storage/exp_soft/tuc/include I use also jupyterhub (with cluster tab enabled) as a user interface to the cluster. After the installation of python and some dependencies mpich and openmpi are also installed in the system directories. -- mpirun --allow-run-as-root --mca shmem_base_verbose 100 ... [se01.grid.tuc.gr:19607] mca: base: components_register: registering framework shmem components [se01.grid.tuc.gr:19607] mca: base: components_register: found loaded component sysv [se01.grid.tuc.gr:19607] mca: base: components_register: component sysv register function successful [se01.grid.tuc.gr:19607] mca: base: components_register: found loaded component posix [se01.grid.tuc.gr:19607] mca: base: components_register: component posix register function successful [se01.grid.tuc.gr:19607] mca: base: components_register: found loaded component mmap [se01.grid.tuc.gr:19607] mca: base: components_register: component mmap register function successful [se01.grid.tuc.gr:19607] mca: base: components_open: opening shmem components [se01.grid.tuc.gr:19607] mca: base: components_open: found loaded component sysv [se01.grid.tuc.gr:19607] mca: base: components_open: component sysv open function successful [se01.grid.tuc.gr:19607] mca: base: components_open: found loaded component posix [se01.grid.tuc.gr:19607] mca: base: components_open: component posix open function successful [se01.grid.tuc.gr:19607] mca: base: components_open: found loaded component mmap [se01.grid.tuc.gr:19607] mca: base: components_open: component mmap open function successful [se01.grid.tuc.gr:19607] shmem: base: runtime_query: Auto-selecting shmem components [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Querying component (run-time) [sysv] [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Query of component [sysv] set priority to 30 [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Querying component (run-time) [posix] [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Query of component [posix] set priority to 40 [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Querying component (run-time) [mmap] [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Query of component [mmap] set priority to 50 [se01.grid.tuc.gr:19607] shmem: base: runtime_query: (shmem) Selected component [mmap] [se01.grid.tuc.gr:19607] mca: base: close: unloading component sysv [se01.grid.tuc.gr:19607] mca: base: close: unloading component posix [se01.grid.tuc.gr:19607] shmem: base: best_runnable_component_name: Searching for best runnable component. [se01.grid.tuc.gr:19607] shmem: base: best_runnable_component_name: Found best runnable component: (mmap). -- mpirun was unable to find the specified executable file, and therefore did not launch the job. This error was first reported for process rank 0; it may have occurred for other processes as well. NOTE: A common cause for this error is misspelling a mpirun command line parameter option (remember that mpirun interprets the first unrecognized command line token as the executable). Node: se01 Executable: ... -
