[OMPI users] MPI_Allgather/MPI_Allgatherv with odd-number nodes

[孟宪军]

Dear all,

I am doing some tests using MPI_Allgatherv function recently. After these
tests, I found a wield problem of if.

When I wanted to use the MPI_Allgatherv function to gather a large number of
data from some processes(for example, 2GB data per process). If the number
of processes was even number, the function worked well and my network card
can receive *and *send data at the same time at his max speed; But if the
number of processes was odd number, the problem came, I found my network
card can only receive *or *send data at the same time at the max speed.

*My sample enviroments*: Openmpi 1.4.3, Linux 2.6.32

*My source codes*:

int main(int argc, char **argv)
{
    char *psend_buf, *precv_buf;
    int rank;
    int process_cnt;
    int repeat_time;

    int *pele_cnt;
    int *pdipal;

    long buf_size;      // Assume the long keyword is 64-bits width

    buf_size = 2000;    // 2000MB
    repeat_time = 1;

    MPI_Init(&argc, &argv);
    MPI_Datatype mpi_meta;

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &process_cnt);

    precv_buf = (char*)malloc(buf_size * process_cnt * 1024 * 1024);
    psend_buf = (char*)malloc(buf_size * 1024 * 1024);
    memset(precv_buf, 0, buf_size * process_cnt * 1024 * 1024);
    memset(psend_buf, 1, buf_size * 1024 * 1024);

    // new data type: 1MB per unit
    MPI_Type_contiguous(1024 * 1024, MPI_CHAR, &mpi_meta);
    MPI_Type_commit(&mpi_meta);

    pele_cnt = (int*)malloc(sizeof(int) * process_cnt);
    pdipal = (int*)malloc(sizeof(int) * process_cnt);

    for (int i = 0; i < process_cnt; i++)
    {
        pele_cnt[i] = buf_size;
        pdipal[i] = i * buf_size;
    }

    for (int i = 0; i < repeat_time; i++)
    {
        MPI_Allgatherv(psend_buf, buf_size, mpi_meta, precv_buf, pele_cnt,
pdipal, mpi_meta, \
                MPI_COMM_WORLD);
    }

    printf("rank %d, used time = %.3f\n", rank, totle_time);

    free(psend_buf);
    free(precv_buf);

    free(pele_cnt);
    free(pdipal);

    MPI_Type_free(&mpi_meta);
    MPI_Finalize();
}


I guess the problem is because of the implementation of the algorithm of
MPI_Allgather. Did anybody meet the same problem and have any suggestions
for me? Thanks

Best Regards
Xianjun