Hi, I will use a small part of C++ code to demonstrate my problem during
shuffling. Assume that each slave has to shuffle some unsigned char array
defined as *unsigned char* data *within some intracommunicator.
*unsigned lineSize = 100;*
*unsigned long long no_keys = 10;*
*int bytes_send_count = (int)no_keys*lineSize; *
*unsigned int commSize = (unsigned)comm.Get_size();*
*int* recv_counts = new int[commSize];*
*int* displs = new int[commSize];*
*//Shuffle amount of data*
*comm.Allgather(&bytes_send_count, 1, MPI::INT, recv_counts, 1, MPI::INT); *
*unsigned long long total = 0; *
*for(unsigned int i = 0; i < commSize; i++){ *
* //Update the displacements*
* displs[i] = total; *
* //...and the total count*
* total += recv_counts[i]; *
*}*
*unsigned char* recv_buf = new unsigned char[total];*
*//Print data to be sent from rank == 1*
*if(rank == 1){*
* for(int l=0; l<bytes_send_count;l++){*
* printf("%d: Data to be sent is %d\n", rank, data[l]);*
* }*
*}*
*//Shuffle actual data*
*comm.Allgatherv(&data, bytes_send_count, MPI::UNSIGNED_CHAR, recv_buf,
recv_counts, displs, MPI::UNSIGNED_CHAR);*
*//Check the first portion of the received data*
*if(rank == 1){*
* for(int l=0; l<recv_counts[0]; l++){*
* printf("%d: Data received from myself is %d\n", rank, recv_buf[l]);*
* }*
*}*
My problem is that the printf() that checks what is about to be sent from
node 1 and what is actually received from node 1 by itself print different
things that don't match. Based on my study of Allgatherv() I think that the
sizes of the received blocks and the displacements are computed correctly.
I don't think I need MPI_IN_PLACE since the input and output buffers are
supposed to be different.
Can you help me identify the problem?
I am using Open MPI 2.1.2 and testing on a single computer with 7 MPI
processes. The ompi_info is the attached file.
Package: Open MPI ubuntu@ip-172-31-30-250 Distribution
Open MPI: 2.1.2
Open MPI repo revision: v2.1.1-188-g6157ed8
Open MPI release date: Sep 20, 2017
Open RTE: 2.1.2
Open RTE repo revision: v2.1.1-188-g6157ed8
Open RTE release date: Sep 20, 2017
OPAL: 2.1.2
OPAL repo revision: v2.1.1-188-g6157ed8
OPAL release date: Sep 20, 2017
MPI API: 3.1.0
Ident string: 2.1.2
Prefix: /usr/local
Configured architecture: x86_64-unknown-linux-gnu
Configure host: ip-172-31-30-250
Configured by: ubuntu
Configured on: Sun Nov 19 22:08:57 UTC 2017
Configure host: ip-172-31-30-250
Built by: root
Built on: Sun Nov 19 22:18:49 UTC 2017
Built host: ip-172-31-30-250
C bindings: yes
C++ bindings: yes
Fort mpif.h: yes (all)
Fort use mpi: yes (full: ignore TKR)
Fort use mpi size: deprecated-ompi-info-value
Fort use mpi_f08: yes
Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
limitations in the gfortran compiler, does not
support the following: array subsections, direct
passthru (where possible) to underlying Open MPI's
C functionality
Fort mpi_f08 subarrays: no
Java bindings: no
Wrapper compiler rpath: runpath
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 5.4.0
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fort compiler: gfortran
Fort compiler abs: /usr/bin/gfortran
Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
Fort 08 assumed shape: yes
Fort optional args: yes
Fort INTERFACE: yes
Fort ISO_FORTRAN_ENV: yes
Fort STORAGE_SIZE: yes
Fort BIND(C) (all): yes
Fort ISO_C_BINDING: yes
Fort SUBROUTINE BIND(C): yes
Fort TYPE,BIND(C): yes
Fort T,BIND(C,name="a"): yes
Fort PRIVATE: yes
Fort PROTECTED: yes
Fort ABSTRACT: yes
Fort ASYNCHRONOUS: yes
Fort PROCEDURE: yes
Fort USE...ONLY: yes
Fort C_FUNLOC: yes
Fort f08 using wrappers: yes
Fort MPI_SIZEOF: yes
C profiling: yes
C++ profiling: yes
Fort mpif.h profiling: yes
Fort use mpi profiling: yes
Fort use mpi_f08 prof: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: no, OPAL support: yes,
OMPI progress: no, ORTE progress: yes, Event lib:
yes)
Sparse Groups: no
Internal debug support: yes
MPI interface warnings: yes
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
dl support: yes
Heterogeneous support: no
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: native
Symbol vis. support: yes
Host topology support: yes
MPI extensions: affinity, cuda
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA btl: sm (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA btl: openib (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA btl: self (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA event: libevent2022 (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA hwloc: hwloc1112 (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
v2.1.2)
MCA pmix: pmix112 (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v2.1.2)
MCA sec: basic (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA dfs: test (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA dfs: app (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA dfs: orted (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
v2.1.2)
MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
v2.1.2)
MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
v2.1.2)
MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
v2.1.2)
MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA ess: env (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
v2.1.2)
MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA iof: mr_orted (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA iof: mr_hnp (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA notifier: syslog (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA odls: default (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA oob: usock (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA oob: ud (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA rmaps: staged (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rml: oob (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA routed: radix (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA routed: binomial (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA routed: direct (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA routed: debruijn (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rtc: freq (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA state: dvm (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA state: staged_orted (MCA v2.1.0, API v1.0.0, Component
v2.1.2)
MCA state: orted (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA state: novm (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA state: tool (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA state: staged_hnp (MCA v2.1.0, API v1.0.0, Component
v2.1.2)
MCA state: app (MCA v2.1.0, API v1.0.0, Component v2.1.2)
MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: self (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA fcoll: static (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA io: romio314 (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v2.1.2)
MCA pml: v (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v2.1.2)
MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v2.1.2)
MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
v2.1.2)
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
