Hi,
It looks like there is a problem in OpenMPI 1.0.2 with how MPI_COMM_SELF
attributes callback functions are handled by MPI_Finalize().
The following C program register a callback function associated with the
MPI_COMM_SELF communicator to be called during the first steps of
MPI_Finalize(). As shown in this example, this can be used to make sure that
global MPI_Datatype variables associated to global datatypes are freed by
calling MPI_Type_free() before program exit (and thus preventing ugly memory
leaks/outstanding allocations when run in valgrind for example). This mechanism
is used by the library I'm working on as well as by PetSc library.
The program works by taking advantage of the MPI-2 Standard Section 4.8
"Allowing User Function at Process Termination". As it says, the MPI_Finalize()
function calls the delete callback associated to the MPI_COMM_SELF attribute
"before any other part of MPI are affected". It also says that "calling
MPI_Finalized() will return false in any of these callback functions".
Section 4.9 of the MPI-2 Standard: "Determining Whether MPI Has Finished"
moreover says that it can be determined if MPI is active by calling
MPI_Finalized(). It also reaffirm that MPI is active in the callback functions
invoked by MPI_Finalize().
I think that an "active" MPI library here means that basic MPI functions like
MPI_Type_free() can be called.
The following small program therefore seems to conform to the MPI standard.
However where I run it (compiled with OpenMPI 1.0.2 mpicc), I get the following
message:
*** An error occurred in MPI_Type_free
*** after MPI was finalized
*** MPI_ERRORS_ARE_FATAL (goodbye)
Note that this program works well with mpich2.
Please have a look at this problem.
Thanks,
Martin Audet
#include <assert.h>
#include <stddef.h>
#include <mpi.h>
static int attr_delete_function(MPI_Comm p_comm, int p_keyval, void
*p_attribute_val, void * p_extra_state)
{
assert(p_attribute_val != NULL);
/* Get a reference on the datatype received. */
MPI_Datatype *const cur_datatype = (MPI_Datatype *)(p_attribute_val);
/* Free it if non null. */
if (*cur_datatype != MPI_DATATYPE_NULL) {
MPI_Type_free(cur_datatype);
assert(*cur_datatype == MPI_DATATYPE_NULL);
}
return MPI_SUCCESS;
}
/* If p_datatype refer to a non null MPI datatype, this function will register
a callback */
/* function to free p_datatype and set it to MPI_DATATYPE_NULL. This callback
will be */
/* called during the first steps of the MPI_Finalize() function when the state
of the MPI */
/* library still allows MPI functions to be called. This is done by
associating an */
/* attribute to the MPI_COMM_SELF communicator as allowed by the MPI 2
standard (section 4.8). */
static void add_type_free_callback(MPI_Datatype *p_datatype)
{
int keyval;
assert(p_datatype != NULL);
/* First create the keyval. */
/* No callback function will be called when MPI_COMM_SELF is duplicated */
/* and attr_delete_function() will be called when MPI_COMM_SELF is */
/* freed (e.g. during MPI_Finalize()). */
/* Since many callback can be associated with MPI_COMM_SELF to free many */
/* datatypes, a new keyval has to be created every time. */
MPI_Keyval_create(MPI_NULL_COPY_FN, &attr_delete_function, &keyval, NULL);
/* Then associate this keyval to MPI_COMM_SELF and make sure the pointer */
/* to the datatype p_datatype is passed to the callback. */
MPI_Attr_put(MPI_COMM_SELF, keyval, p_datatype);
/* Free the keyval because it is no longer needed. */
MPI_Keyval_free(&keyval);
}
typedef struct {
short ss;
int ii;
} glb_struct_t;
MPI_Datatype glb_dtype = MPI_DATATYPE_NULL;
static void calc_glb_dtype(void)
{
const int NB_MEM = 3;
static int len_tbl[3] = { 1, 1,
1 };
static MPI_Aint disp_tbl[3] = { offsetof(glb_struct_t, ss),
offsetof(glb_struct_t, ii), sizeof(glb_struct_t) };
static MPI_Datatype type_tbl[3] = { MPI_SHORT, MPI_INT,
MPI_UB };
MPI_Type_struct(NB_MEM, len_tbl, disp_tbl, type_tbl, &glb_dtype);
MPI_Type_commit(&glb_dtype);
add_type_free_callback(&glb_dtype);
}
int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
calc_glb_dtype();
MPI_Finalize();
return 0;
}
