Hi, I am attempting to debug a memory corruption in an mpi program using
valgrind. Howver, when I run with valgrind I get semi-random segfaults and
valgrind messages with the openmpi library. Here is an example of such a
seg fault:
==6153==
==6153== Invalid read of size 8
==6153== at 0x19102EA0: (within
/usr/lib/openmpi/lib/openmpi/mca_btl_sm.so)
==6153== by 0x182ABACB: (within
/usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
==6153== by 0x182A3040: (within
/usr/lib/openmpi/lib/openmpi/mca_pml_ob1.so)
==6153== by 0xB425DD3: PMPI_Isend (in
/usr/lib/openmpi/lib/libmpi.so.0.0.0)
==6153== by 0x7B83DA8: int Uintah::SFC<double>::MergeExchange<unsigned
char>(int, std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&,
std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&,
std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&) (SFC.h:2989)
==6153== by 0x7B84A8F: void Uintah::SFC<double>::Batchers<unsigned
char>(std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&,
std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&,
std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&) (SFC.h:3730)
==6153== by 0x7B8857B: void Uintah::SFC<double>::Cleanup<unsigned
char>(std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&,
std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&,
std::vector<Uintah::History<unsigned char>,
std::allocator<Uintah::History<unsigned char> > >&) (SFC.h:3695)
==6153== by 0x7B88CC6: void Uintah::SFC<double>::Parallel0<3, unsigned
char>() (SFC.h:2928)
==6153== by 0x7C00AAB: void Uintah::SFC<double>::Parallel<3, unsigned
char>() (SFC.h:1108)
==6153== by 0x7C0EF39: void Uintah::SFC<double>::GenerateDim<3>(int)
(SFC.h:694)
==6153== by 0x7C0F0F2: Uintah::SFC<double>::GenerateCurve(int)
(SFC.h:670)
==6153== by 0x7B30CAC:
Uintah::DynamicLoadBalancer::useSFC(Uintah::Handle<Uintah::Level> const&,
int*) (DynamicLoadBalancer.cc:429)
==6153== Address 0x10 is not stack'd, malloc'd or (recently) free'd
^G^G^GThread "main"(pid 6153) caught signal SIGSEGV at address (nil)
(segmentation violation)
Looking at the code for our isend at SFC.h:298 does not seem to have any
errors:
=============================================
MergeInfo<BITS> myinfo,theirinfo;
MPI_Request srequest, rrequest;
MPI_Status status;
myinfo.n=n;
if(n!=0)
{
myinfo.min=sendbuf[0].bits;
myinfo.max=sendbuf[n-1].bits;
}
//cout << rank << " n:" << n << " min:" << (int)myinfo.min << "max:" <<
(int)myinfo.max << endl;
MPI_Isend(&myinfo,sizeof(MergeInfo<BITS>),MPI_BYTE,to,0,Comm,&srequest);
==============================================
myinfo is a struct located on the stack, to is the rank of the processor
that the message is being sent to, and srequest is also on the stack. When
I don't run with valgrind my program runs past this point just fine.
I am currently using openmpi 1.3 from the debian unstable branch. I also
see the same type of segfault in a different portion of the code involving
an MPI_Allgather which can be seen below:
==============================================
==22736== Use of uninitialised value of size 8
==22736== at 0x19104775: mca_btl_sm_component_progress (opal_list.h:322)
==22736== by 0x1382CE09: opal_progress (opal_progress.c:207)
==22736== by 0xB404264: ompi_request_default_wait_all (condition.h:99)
==22736== by 0x1A1ADC16: ompi_coll_tuned_sendrecv_actual
(coll_tuned_util.c:55)
==22736== by 0x1A1B61E1: ompi_coll_tuned_allgatherv_intra_bruck
(coll_tuned_util.h:60)
==22736== by 0xB418B2E: PMPI_Allgatherv (pallgatherv.c:121)
==22736== by 0x646CCF7: Uintah::Level::setBCTypes() (Level.cc:728)
==22736== by 0x646D823: Uintah::Level::finalizeLevel() (Level.cc:537)
==22736== by 0x6465457:
Uintah::Grid::problemSetup(Uintah::Handle<Uintah::ProblemSpec> const&,
Uintah::ProcessorGroup const*, bool) (Grid.cc:866)
==22736== by 0x8345759: Uintah::SimulationController::gridSetup()
(SimulationController.cc:243)
==22736== by 0x834F418: Uintah::AMRSimulationController::run()
(AMRSimulationController.cc:117)
==22736== by 0x4089AE: main (sus.cc:629)
==22736==
==22736== Invalid read of size 8
==22736== at 0x19104775: mca_btl_sm_component_progress (opal_list.h:322)
==22736== by 0x1382CE09: opal_progress (opal_progress.c:207)
==22736== by 0xB404264: ompi_request_default_wait_all (condition.h:99)
==22736== by 0x1A1ADC16: ompi_coll_tuned_sendrecv_actual
(coll_tuned_util.c:55)
==22736== by 0x1A1B61E1: ompi_coll_tuned_allgatherv_intra_bruck
(coll_tuned_util.h:60)
==22736== by 0xB418B2E: PMPI_Allgatherv (pallgatherv.c:121)
==22736== by 0x646CCF7: Uintah::Level::setBCTypes() (Level.cc:728)
==22736== by 0x646D823: Uintah::Level::finalizeLevel() (Level.cc:537)
==22736== by 0x6465457:
Uintah::Grid::problemSetup(Uintah::Handle<Uintah::ProblemSpec> const&,
Uintah::ProcessorGroup const*, bool) (Grid.cc:866)
==22736== by 0x8345759: Uintah::SimulationController::gridSetup()
(SimulationController.cc:243)
==22736== by 0x834F418: Uintah::AMRSimulationController::run()
(AMRSimulationController.cc:117)
==22736== by 0x4089AE: main (sus.cc:629)
================================================================
Are these problems with openmpi and is there any known work arounds?
Thanks,
Justin
