[OMPI users] Openmpi failure on dual quad linux

John R. Cary Sat, 10 Oct 2009 12:09:52 -0400

We have been getting hangs and failures of openmpi-1.3.X
on an 8-core FC11 box.  Details:


Machine:

Linux octet.carys.home 2.6.30.5-43.fc11.x86_64 #1 SMP Thu Aug 2721:39:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux


with 2 quad-core cpus.

Compiler:

g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2)

The code is later in this email.


OpenMPI configured with

../configure --prefix=/usr/local/openmpi-1.3.3--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3/lib


used to compile and execute:

/usr/local/openmpi-1.3.3/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.3.3/bin/mpirun -np 8 ompi1.3.3-bug

hangs on various steps: 10, 39, 125.  Top shows the code
using all resources:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+COMMAND5716 cary 20 0 98.6m 4396 2940 R 100.3 0.0 0:33.58ompi1.3.3-bug5707 cary 20 0 98.7m 4592 3060 R 99.9 0.0 0:33.57ompi1.3.3-bug5710 cary 20 0 98.6m 4388 2932 R 99.9 0.0 0:33.61ompi1.3.3-bug5713 cary 20 0 98.6m 4392 2936 R 99.9 0.0 0:33.61ompi1.3.3-bug5719 cary 20 0 98.6m 4396 2940 R 99.9 0.0 0:33.60ompi1.3.3-bug5722 cary 20 0 98.6m 4384 2928 R 99.9 0.0 0:33.61ompi1.3.3-bug5728 cary 20 0 98.6m 4408 2952 R 99.9 0.0 0:33.53ompi1.3.3-bug5725 cary 20 0 98.6m 4404 2948 R 99.6 0.0 0:33.57 ompi1.3.3-bug


OpenMPI configured with

../configure --prefix=/usr/local/openmpi-1.2.9--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.2.9/lib


used to compile and execute:

/usr/local/openmpi-1.2.9/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.2.9/bin/mpirun -np 8 ompi1.3.3-bug

works just fine.


So then I started looking for unitialized variables.  I rebuilt
OpenMPI, configured with

../configure --prefix=/usr/local/openmpi-1.3.3-valg--with-valgrind=/contrib/valgrind--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3-valg/lib


and used to to compile and execute:

/usr/local/openmpi-1.3.3-valg/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind ompi1.3.3-bug


and one can get valgrind to segfault, as shown at the very end of this
email.  Other times it hangs on step 1 or another step.

John Cary

THE CODE

$ cat ompi1.3.3-bug.cxx


/**
* A simple test program to demonstrate a problem in OpenMPI 1.3
*/

// mpi includes
#include <mpi.h>

// std includes
#include <iostream>
#include <vector>

// useful hashdefine
#define ARRAY_SIZE 250

/**
* Main driver
*/
int main(int argc, char** argv) {
// Initialize MPI
 MPI_Init(&argc, &argv);

 int rk, sz;
 MPI_Comm_rank(MPI_COMM_WORLD, &rk);
 MPI_Comm_size(MPI_COMM_WORLD, &sz);

// Create some data to pass around
 std::vector<double> d(ARRAY_SIZE);

// Initialize to some values if we aren't rank 0
 if ( rk )
   for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
     d[i] = 2*i + 1;

// Loop until this breaks
 unsigned t = 0;
 while ( 1 ) {
   MPI_Status s;
   if ( rk )
     MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
   else
     for ( int i = 1; i < sz; ++i )
       MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
   MPI_Barrier(MPI_COMM_WORLD);
   std::cout << "Transmission " << ++t << " completed." << std::endl;
 }

// Finalize MPI
 MPI_Finalize();
}

THE VALGRIND RUN

octet.cary$ /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrindompi1.3.3-bug

==902== Memcheck, a memory error detector
==902== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==902== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==902== Command: ompi1.3.3-bug
==902==
==905== Memcheck, a memory error detector
==905== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==905== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==905== Command: ompi1.3.3-bug
==905==
==908== Memcheck, a memory error detector
==908== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==908== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==908== Command: ompi1.3.3-bug
==908==
==911== Memcheck, a memory error detector
==911== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==911== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==911== Command: ompi1.3.3-bug
==911==
==914== Memcheck, a memory error detector
==914== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==914== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==914== Command: ompi1.3.3-bug
==914==
==917== Memcheck, a memory error detector
==917== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==917== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==917== Command: ompi1.3.3-bug
==917==
==920== Memcheck, a memory error detector
==920== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==920== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==920== Command: ompi1.3.3-bug
==920==
==923== Memcheck, a memory error detector
==923== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==923== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==923== Command: ompi1.3.3-bug
==923==

==917== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==902== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==905== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==914== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==911== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==920== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==923== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe==908== Warning: client syscall munmap tried to modify addresses0xffffffffffffffff-0xffe

==902== Conditional jump or move depends on uninitialised value(s)

==902== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==902== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==902== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==902== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==902== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==902==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==902== Conditional jump or move depends on uninitialised value(s)

==902== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==902== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==902== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==902== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==902== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==902==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)

==911== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==911== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==911== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==911== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==911== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==911==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==902== Conditional jump or move depends on uninitialised value(s)

==902== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==902== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==902== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==902== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==902== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==914== Conditional jump or move depends on uninitialised value(s)
==902==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==

==914== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==914== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==914== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==914== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==914== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==914==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917== Conditional jump or move depends on uninitialised value(s)

==917== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==917== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==917== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==917== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==917== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==917==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==911== Conditional jump or move depends on uninitialised value(s)

==911== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==911== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==911== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==911== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==911== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==911==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==914== Conditional jump or move depends on uninitialised value(s)

==914== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==914== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==914== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==914== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==914== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==914==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==908== Conditional jump or move depends on uninitialised value(s)

==908== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==908== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==908== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==908== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==908== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==908==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==902== Conditional jump or move depends on uninitialised value(s)

==902== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==902== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==902== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==902== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==902== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==902==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)

==911== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==911== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923== Conditional jump or move depends on uninitialised value(s)

==911== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==911== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==911== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==911== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==911==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==

==923== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==923== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)

==920== Conditional jump or move depends on uninitialised value(s)

==923== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==923== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==923== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==

==920== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==920== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==920== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==920== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==920== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==920==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==914== Conditional jump or move depends on uninitialised value(s)

==914== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==914== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==914== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==914== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==914== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==914==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917== Conditional jump or move depends on uninitialised value(s)

==917== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==917== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==917== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==917== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==917== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==917==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==917== Conditional jump or move depends on uninitialised value(s)
==908== Conditional jump or move depends on uninitialised value(s)

==908== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==908== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==908== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==908== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==908== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==908==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==

==917== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==917== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==917== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==917== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==917== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==911== Conditional jump or move depends on uninitialised value(s)

==911== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==911== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==911== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==911== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==911== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==911==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==

==917== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==917==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914== Conditional jump or move depends on uninitialised value(s)

==914== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==914== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==914== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==914== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==914== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==914==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917==
==923== Conditional jump or move depends on uninitialised value(s)

==923== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==923== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==923== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==923== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==923== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)

==920== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==920== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==920== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==920== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==920== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==920==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==908== Conditional jump or move depends on uninitialised value(s)

==908== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==908== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==908== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==908== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==908== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==908==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==923== Conditional jump or move depends on uninitialised value(s)

==923== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==923== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==920== Conditional jump or move depends on uninitialised value(s)

==920== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==920== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==923== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==923== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==923== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==923== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==

==920== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==920== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==920== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==920==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)

==905== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==905== by 0x8C1F94D: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==905== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==905== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==905== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==905==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==917== Conditional jump or move depends on uninitialised value(s)

==917== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==917== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==917== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==917== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==917== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==917==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==908== Conditional jump or move depends on uninitialised value(s)

==908== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==908== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==908== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==908== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==908== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==908==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==905== Conditional jump or move depends on uninitialised value(s)

==905== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==905== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x8C1FA22: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==905== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==905== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==905== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==905==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==923== Conditional jump or move depends on uninitialised value(s)

==923== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==923== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==923== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==923== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==923== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)

==920== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==920== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==920== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==920== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==920== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==920==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)

==905== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==905== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x8C1FA93: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==905== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==905== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==905== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==905==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==917== Conditional jump or move depends on uninitialised value(s)

==917== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==917== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==917== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==917== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==917== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==917== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==917==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==908== Conditional jump or move depends on uninitialised value(s)

==908== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==908== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==908== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==908== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==908== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==908== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==908==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==923== Conditional jump or move depends on uninitialised value(s)

==923== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==923== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==923== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==923== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==923== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)

==920== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==920== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==920== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==920== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==920== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==920== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==920==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)

==905== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==905== by 0x8C1F0FC: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==905== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==905== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==905== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==905==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==905== Conditional jump or move depends on uninitialised value(s)

==905== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==905== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==905== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==905== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==905== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==905== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==905==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==902== Conditional jump or move depends on uninitialised value(s)

==902== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==902== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==902== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==902== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==902== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==902== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==902==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)

==911== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==911== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==911== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==911== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==911== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==911== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==911==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==914== Conditional jump or move depends on uninitialised value(s)

==914== at 0x7BDFD64: mca_mpool_sm_alloc (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)==914== by 0x4E3EBDF: ompi_free_list_grow (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x4E3EC5F: ompi_free_list_resize (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x8C1F630: mca_btl_sm_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==914== by 0x8816759: mca_bml_r2_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)==914== by 0x8400274: mca_pml_ob1_add_procs (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)==914== by 0x4E5B048: ompi_mpi_init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==914== by 0x4E79F65: PMPI_Init (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==914==    by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
[octet:00923] *** Process received signal ***
[octet:00923] Signal: Segmentation fault (11)
[octet:00923] Signal code: Address not mapped (1)
[octet:00923] Failing at address: 0x40
[octet:00923] [ 0] /lib64/libpthread.so.0 [0x3825c0eee0]

[octet:00923] [ 1]/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so [0x8c21009][octet:00923] [ 2]/usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0(opal_progress+0x5a)[0x5324c9a]

[octet:00923] [ 3] /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0 [0x4e59d45]

[octet:00923] [ 4]/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9ccad2a][octet:00923] [ 5]/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9cd349e][octet:00923] [ 6]/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0(MPI_Barrier+0x72) [0x4e6fa32]

[octet:00923] [ 7] ompi1.3.3-bug(main+0x1ee) [0x409152]
[octet:00923] [ 8] /lib64/libc.so.6(__libc_start_main+0xfd) [0x382501ea2d]
[octet:00923] [ 9] ompi1.3.3-bug [0x408ea9]
[octet:00923] *** End of error message ***
==923==
==923== Process terminating with default action of signal 11 (SIGSEGV)
==923==  Access not within mapped region at address 0x40

==923== at 0x8C21009: mca_btl_sm_component_progress (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)==923== by 0x5324C99: opal_progress (in/usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0.0.0)==923== by 0x4E59D44: ompi_request_default_wait_all (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)==923== by 0x9CCAD29: ompi_coll_tuned_sendrecv_actual (in/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)==923== by 0x9CD349D: ompi_coll_tuned_barrier_intra_recursivedoubling(in /usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)==923== by 0x4E6FA31: PMPI_Barrier (in/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)

==923==    by 0x409151: main (in /home/research/cary/ompi1.3.3-bug)
==923==  If you believe this happened as a result of a stack
==923==  overflow in your program's main thread (unlikely but
==923==  possible), you can try to increase the size of the
==923==  main thread stack using the --main-stacksize= flag.
==923==  The main thread stack size used in this run was 10485760.
==923==
==923== HEAP SUMMARY:
==923==     in use at exit: 2,062,803 bytes in 2,652 blocks

==923== total heap usage: 10,244 allocs, 7,592 frees, 15,824,029 bytesallocated

==923==
==923== LEAK SUMMARY:
==923==    definitely lost: 1,700 bytes in 29 blocks
==923==    indirectly lost: 2,400 bytes in 1 blocks
==923==      possibly lost: 2,384 bytes in 22 blocks
==923==    still reachable: 2,056,319 bytes in 2,600 blocks
==923==         suppressed: 0 bytes in 0 blocks
==923== Rerun with --leak-check=full to see details of leaked memory
==923==
==923== For counts of detected and suppressed errors, rerun with: -v
==923== Use --track-origins=yes to see where uninitialised values come from
==923== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 7 from 7)
--------------------------------------------------------------------------

mpirun noticed that process rank 7 with PID 923 on node octet.carys.homeexited on signal 11 (Segmentation fault).

--------------------------------------------------------------------------


--
Tech-X Corp., 5621 Arapahoe Ave, Suite A, Boulder CO 80303
c...@txcorp.com, p 303-448-0727, f 303-448-7756, NEW CELL 303-881-8572

[OMPI users] Openmpi failure on dual quad linux

Reply via email to