[OMPI users] OpenMPI 1.2.5 race condition / core dump with MPI_Reduce and MPI_Gather
Hi,
I ran into a bug when running a few microbenchmarks for OpenMPI. I had
thrown in Reduce and Gather for sanity checking, but OpenMPI crashed
when running those operations. Usually, this would happen when I reached
around 12-16 nodes.
My current crash-test code looks like this (I've removed a few lines
that were commented out):
--- snip-
#include
#include
#include
#include
#include
int main(int argc, char *argv[])
{
int rank, size, count = 1;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
if (argc > 1)
count = atoi(argv[1]);
int n, i;
// Just make sure we have plenty of buffer for any operation
int *sbuf = malloc(sizeof(int) * 2 * count);
int *rbuf = malloc(sizeof(int) * 2 * count);
assert(sbuf);
assert(rbuf);
for (n = 1; n <= 1; n += 100) {
printf("N = %d\n", n);
fflush(stdout);
for (i = 0; i < n; i++) {
MPI_Reduce(sbuf, rbuf, count, MPI_INT, MPI_SUM, 0,
MPI_COMM_WORLD);
}
MPI_Barrier(MPI_COMM_WORLD);
printf(" -- DONE\n");
fflush(stdout);
MPI_Barrier(MPI_COMM_WORLD);
}
MPI_Finalize();
return 0;
}
--- snip-
I can usually trigger a crash with count=1, and n=1000 using 16+ nodes,
but I can also trigger it with 44 nodes and larger packets (around 32k
ints I think). I can also crash it on a single host using 19 processes,
but then it usually doesn't crash until I reach somewhere between
1200-3000 iterations. Gather seems to have the same problems as Reduce.
The output from running gdb on the coredump looks like this:
--- snip-
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `./ompi-crash2'.
Program terminated with signal 11, Segmentation fault.
#0 0x00434184 in sysconf () from /lib/tls/libc.so.6
#0 0x00434184 in sysconf () from /lib/tls/libc.so.6
#1 0xb7e78b59 in _int_malloc () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#2 0xb7e799ce in malloc () from /home/johnm/local/ompi/lib/libopen-pal.so.0
#3 0xb7f04852 in ompi_free_list_grow () from
/home/johnm/local/ompi/lib/libmpi.so.0
#4 0xb7d74e70 in mca_btl_tcp_endpoint_recv_handler () from
/home/johnm/local/ompi/lib/openmpi/mca_btl_tcp.so
#5 0xb7e62b44 in opal_event_base_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#6 0xb7e62cff in opal_event_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#7 0xb7e5d284 in opal_progress () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#8 0xb7d74f08 in mca_btl_tcp_endpoint_recv_handler () from
/home/johnm/local/ompi/lib/openmpi/mca_btl_tcp.so
#9 0xb7e62b44 in opal_event_base_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#10 0xb7e62cff in opal_event_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#11 0xb7e5d284 in opal_progress () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#12 0xb7d74f08 in mca_btl_tcp_endpoint_recv_handler () from
/home/johnm/local/ompi/lib/openmpi/mca_btl_tcp.so
#13 0xb7e62b44 in opal_event_base_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#14 0xb7e62cff in opal_event_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
... and then continues until...
#1356848 0xb7e5d284 in opal_progress () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356849 0xb7d8f389 in mca_pml_ob1_recv_frag_match () from
/home/johnm/local/ompi/lib/openmpi/mca_pml_ob1.so
#1356850 0xb7d74a7d in mca_btl_tcp_endpoint_recv_handler () from
/home/johnm/local/ompi/lib/openmpi/mca_btl_tcp.so
#1356851 0xb7e62b44 in opal_event_base_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356852 0xb7e62cff in opal_event_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356853 0xb7e5d284 in opal_progress () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356854 0xb7d8f389 in mca_pml_ob1_recv_frag_match () from
/home/johnm/local/ompi/lib/openmpi/mca_pml_ob1.so
#1356855 0xb7d74a7d in mca_btl_tcp_endpoint_recv_handler () from
/home/johnm/local/ompi/lib/openmpi/mca_btl_tcp.so
#1356856 0xb7e62b44 in opal_event_base_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356857 0xb7e62cff in opal_event_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356858 0xb7e5d284 in opal_progress () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356859 0xb7d8f389 in mca_pml_ob1_recv_frag_match () from
/home/johnm/local/ompi/lib/openmpi/mca_pml_ob1.so
#1356860 0xb7d74a7d in mca_btl_tcp_endpoint_recv_handler () from
/home/johnm/local/ompi/lib/openmpi/mca_btl_tcp.so
#1356861 0xb7e62b44 in opal_event_base_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356862 0xb7e62cff in opal_event_loop () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356863 0xb7e5d284 in opal_progress () from
/home/johnm/local/ompi/lib/libopen-pal.so.0
#1356864 0xb7d8cb69 in mca_pml_ob1_recv () from
/home/johnm/local/ompi/lib/openmpi/mca_pm
Re: [OMPI users] mpi.h macro naming
On Thu, 21 Feb 2008, Jeff Squyres wrote: While I agree that having AC actually define them to a value is a Good Thing (better than just defining it to be empty), I do see the pickle that it has put us in. :-\ I don't see an obvious solution. I do :). Try the attached patch. It sneaks in at the last minute and defines (for example) both SIZEOF_BOOL and OMPI_SIZEOF_BOOL. mpi.h only has the define for OMPI_SIZEOF_BOOL. opal_config.h has both, but a note saying not to use the OMPI_ version. Seems to work. Brian Index: configure.ac === --- configure.ac (revision 17548) +++ configure.ac (working copy) @@ -73,6 +73,29 @@ OMPI_SAVE_VERSION([OPAL], [Open Portable Access Layer], [$srcdir/VERSION], [opal/include/opal/version.h]) + +# Recreate some defines prefixed with OMPI_ so that there are no bare +# autoconf macro defines in mpi.h. Since AC sometimes changes whether +# things are defined as null tokens or an integer result, two projects +# with different versions of AC can cause problems. +if test $ac_cv_header_stdc = yes; then + AC_DEFINE(OMPI_STDC_HEADERS, 1, + [Do not use outside of mpi.h. Define to 1 if you have the ANSI C header files.]) +fi +if test $ac_cv_header_sys_time_h = yes ; then + AC_DEFINE(OMPI_HAVE_SYS_TIME_H, 1, + [Do not use outside of mpi.h. Define to 1 if you have the header file.]) +fi +if test $ac_cv_type_long_long = yes ; then + AC_DEFINE(OMPI_HAVE_LONG_LONG, 1, + [Do not use outside of mpi.h. Define to 1 if the system has the type `long long'.]) dnl ` +fi +AC_DEFINE_UNQUOTED(OMPI_SIZEOF_BOOL, $ac_cv_sizeof_bool, + [Do not use outside of mpi.h. The size of a `bool', as computed by sizeof.]) dnl ` +AC_DEFINE_UNQUOTED(OMPI_SIZEOF_INT, $ac_cv_sizeof_int, + [Do not use outside of mpi.h. The size of a `int', as computed by sizeof.]) dnl ` + + AM_CONFIG_HEADER([opal/include/opal_config.h orte/include/orte_config.h ompi/include/ompi_config.h ompi/include/mpi.h]) # override/fixup the version numbers set by AC_INIT, since on Index: ompi/include/mpi.h.in === --- ompi/include/mpi.h.in (revision 17548) +++ ompi/include/mpi.h.in (working copy) @@ -33,6 +33,21 @@ ompi_config.h must be included before all other files, so this should be good enough */ +/* Define to 1 if you have the ANSI C header files. */ +#undef OMPI_STDC_HEADERS + +/* Define to 1 if you have the header file. */ +#undef OMPI_HAVE_SYS_TIME_H + +/* Define to 1 if the system has the type `long long'. */ +#undef OMPI_HAVE_LONG_LONG + +/* The size of a `bool', as computed by sizeof. */ +#undef OMPI_SIZEOF_BOOL + +/* The size of a `int', as computed by sizeof. */ +#undef OMPI_SIZEOF_INT + /* Whether we have FORTRAN LOGICAL*1 or not */ #undef OMPI_HAVE_FORTRAN_LOGICAL1 @@ -72,18 +87,6 @@ /* Whether we have FORTRAN REAL*8 or not */ #undef OMPI_HAVE_FORTRAN_REAL8 -/* Define to 1 if you have the header file. */ -#undef HAVE_SYS_TIME_H - -/* Define to 1 if the system has the type `long long'. */ -#undef HAVE_LONG_LONG - -/* The size of a `bool', as computed by sizeof. */ -#undef SIZEOF_BOOL - -/* The size of a `int', as computed by sizeof. */ -#undef SIZEOF_INT - /* Type of MPI_Offset -- has to be defined here and typedef'ed later because mpi.h does not get AC SUBST's */ #undef OMPI_MPI_OFFSET_TYPE @@ -111,9 +114,6 @@ /* MPI datatype corresponding to MPI_Offset */ #undef OMPI_OFFSET_DATATYPE -/* Define to 1 if you have the ANSI C header files. */ -#undef STDC_HEADERS - /* Major, minor, and release version of Open MPI */ #undef OMPI_MAJOR_VERSION #undef OMPI_MINOR_VERSION
Re: [OMPI users] mpi.h macro naming
Oops.. forgot to test the C++ bindings, which of course broke. Let me try again and I'll send a better patch. brian On Fri, 22 Feb 2008, Brian W. Barrett wrote: On Thu, 21 Feb 2008, Jeff Squyres wrote: While I agree that having AC actually define them to a value is a Good Thing (better than just defining it to be empty), I do see the pickle that it has put us in. :-\ I don't see an obvious solution. I do :). Try the attached patch. It sneaks in at the last minute and defines (for example) both SIZEOF_BOOL and OMPI_SIZEOF_BOOL. mpi.h only has the define for OMPI_SIZEOF_BOOL. opal_config.h has both, but a note saying not to use the OMPI_ version. Seems to work. Brian
Re: [OMPI users] mpi.h macro naming
A second attempt, this time with less suck :).
Brian
On Fri, 22 Feb 2008, Brian W. Barrett wrote:
Oops.. forgot to test the C++ bindings, which of course broke. Let me
try again and I'll send a better patch.
brian
On Fri, 22 Feb 2008, Brian W. Barrett wrote:
On Thu, 21 Feb 2008, Jeff Squyres wrote:
While I agree that having AC actually define them to a value is a Good
Thing (better than just defining it to be empty), I do see the pickle
that it has put us in. :-\ I don't see an obvious solution.
I do :). Try the attached patch. It sneaks in at the last minute and
defines (for example) both SIZEOF_BOOL and OMPI_SIZEOF_BOOL. mpi.h only has
the define for OMPI_SIZEOF_BOOL. opal_config.h has both, but a note saying
not to use the OMPI_ version. Seems to work.
Brian
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
Index: include/mpi.h.in
===
--- include/mpi.h.in (revision 17548)
+++ include/mpi.h.in (working copy)
@@ -33,6 +33,21 @@
ompi_config.h must be included before all other files, so this
should be good enough */
+/* Define to 1 if you have the ANSI C header files. */
+#undef OMPI_STDC_HEADERS
+
+/* Define to 1 if you have the header file. */
+#undef OMPI_HAVE_SYS_TIME_H
+
+/* Define to 1 if the system has the type `long long'. */
+#undef OMPI_HAVE_LONG_LONG
+
+/* The size of a `bool', as computed by sizeof. */
+#undef OMPI_SIZEOF_BOOL
+
+/* The size of a `int', as computed by sizeof. */
+#undef OMPI_SIZEOF_INT
+
/* Whether we have FORTRAN LOGICAL*1 or not */
#undef OMPI_HAVE_FORTRAN_LOGICAL1
@@ -72,18 +87,6 @@
/* Whether we have FORTRAN REAL*8 or not */
#undef OMPI_HAVE_FORTRAN_REAL8
-/* Define to 1 if you have the header file. */
-#undef HAVE_SYS_TIME_H
-
-/* Define to 1 if the system has the type `long long'. */
-#undef HAVE_LONG_LONG
-
-/* The size of a `bool', as computed by sizeof. */
-#undef SIZEOF_BOOL
-
-/* The size of a `int', as computed by sizeof. */
-#undef SIZEOF_INT
-
/* Type of MPI_Offset -- has to be defined here and typedef'ed later because mpi.h does not get AC SUBST's */
#undef OMPI_MPI_OFFSET_TYPE
@@ -111,9 +114,6 @@
/* MPI datatype corresponding to MPI_Offset */
#undef OMPI_OFFSET_DATATYPE
-/* Define to 1 if you have the ANSI C header files. */
-#undef STDC_HEADERS
-
/* Major, minor, and release version of Open MPI */
#undef OMPI_MAJOR_VERSION
#undef OMPI_MINOR_VERSION
Index: mpi/cxx/comm_inln.h
===
--- mpi/cxx/comm_inln.h (revision 17548)
+++ mpi/cxx/comm_inln.h (working copy)
@@ -648,7 +648,7 @@
void* extra_state, void* attribute_val_in,
void* attribute_val_out, bool& flag)
{
-#if SIZEOF_BOOL != SIZEOF_INT
+#if OMPI_SIZEOF_BOOL != OMPI_SIZEOF_INT
int f = (int)flag;
int ret;
ret = MPI_DUP_FN(oldcomm, comm_keyval, extra_state, attribute_val_in,
