This looks like a problem with the memory allocator. It could be a
genuine problem with Open MPI, or it could be a memory fault in your
application (that happens to dead-end in one of our libraries because
we intercept memory allocation functions). From the stack trace, it
looks like you're in the middle of a complex deallocation of some C++
objects, so I really can't tell (i.e., not in an MPI function at all).
Can you do a few things:
- configure your Open MPI --with-memory-manager=none and see if the
problem goes away. This tells Open MPI to not intercept memory
manager functions, so if you still have the problem, it's more likely
to be a problem in your application than in OMPI.
- run your application through a memory-checking debugger (such as
valgrind) and see if it identifies any memory faults within your code.
- send the additional information for run-time problems listed on
http://www.open-mpi.org/community/help/
Thanks!
On Jan 25, 2006, at 10:39 PM, Glenn Morris wrote:
I tried nightly snapshot 1.1a1r8803 and it said the following. I'm
willing to try and debug this further, but would need some guidance. I
have access to totalview.
Signal:11 info.si_errno:0(Success) si_code:2(SEGV_ACCERR)
Failing at addr:0x97421004
[0] func:/afs/slac.stanford.edu/g/ki/users/gmorris/tmp/
ompi-1.1a1r8803/lib/libopal.so.0 [0x1cc9fa]
[1] func:/lib/tls/libpthread.so.0 [0xfd2f80]
[2] func:/afs/slac.stanford.edu/g/ki/users/gmorris/tmp/
ompi-1.1a1r8803/lib/libopal.so.0(free+0x5e) [0x1cf0a2]
[3] func:./cosmomc(for_deallocate+0x56) [0x80d8806]
[4] func:./cosmomc(for_dealloc_allocatable+0x59) [0x80d886d]
[5] func:./cosmomc(spline_+0x4f2) [0x805a2ea]
[6] func:./cosmomc(cambmain_mp_initsourceinterpolation_+0x243)
[0x8089b65]
[7] func:/afs/slac.stanford.edu/package/intel_tools/compiler8.1/
i386_linux24/ifort/lib/libguide.so(__kmp_invoke_microtask+0x4d)
[0x19f8cd]
[8] func:/afs/slac.stanford.edu/package/intel_tools/compiler8.1/
i386_linux24/ifort/lib/libguide.so(__kmpc_invoke_task_func+0xa2)
[0x18fea6]
[9] func:/afs/slac.stanford.edu/package/intel_tools/compiler8.1/
i386_linux24/ifort/lib/libguide.so(__kmp_internal_fork+0x19b)
[0x1900a1]
[10] func:/afs/slac.stanford.edu/package/intel_tools/compiler8.1/
i386_linux24/ifort/lib/libguide.so(__kmp_fork_call+0x334) [0x18af18]
[11] func:/afs/slac.stanford.edu/package/intel_tools/compiler8.1/
i386_linux24/ifort/lib/libguide.so(__kmpc_fork_call+0x35) [0x19242d]
[12] func:./cosmomc(cambmain_mp_initsourceinterpolation_+0xa8)
[0x80899ca]
[13] func:./cosmomc(cambmain_mp_cmbmain_+0x8a8) [0x8085c34]
[14] func:./cosmomc(camb_mp_camb_getresults_+0x99) [0x80936db]
[15] func:./cosmomc(camb_mp_camb_gettransfers_+0x117) [0x8093387]
[16] func:./cosmomc(cmb_cls_mp_getcls_+0x102) [0x80aa82a]
[17] func:./cosmomc(calclike_mp_getloglikepost_+0x1f9) [0x80b43e9]
[18] func:./cosmomc(calclike_mp_getloglike_+0x23e) [0x80b41d0]
[19] func:./cosmomc(montecarlo_mp_mcmcsample_+0x130) [0x80b68dc]
[20] func:./cosmomc(MAIN__+0x15a3) [0x80b885b]
[21] func:./cosmomc(main+0x20) [0x8059758]
[22] func:/lib/tls/libc.so.6(__libc_start_main+0xda) [0x23c79a]
[23] func:./cosmomc(sinh+0x49) [0x8059611]
*** End of error message ***
1 process killed (possibly by Open MPI)
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
