Yoinks; I have no idea how we missed this. Thanks for bringing it to
our attention.
There's a few subtleties involved in fixing this -- we're internally
debating the correct fix. I'll get back to you when we come to a
conclusion on the Right way to fix it.
________________________________
From: users-boun...@open-mpi.org
[mailto:users-boun...@open-mpi.org] On Behalf Of Audet, Martin
Sent: Monday, April 10, 2006 6:34 PM
To: us...@open-mpi.org
Subject: [OMPI users] Incorrect behavior for attributes attached
toMPI_COMM_SELF.
Hi,
It looks like there is a problem in OpenMPI 1.0.2 with how
MPI_COMM_SELF attributes callback functions are handled by
MPI_Finalize().
The following C program register a callback function associated
with the MPI_COMM_SELF communicator to be called during the first steps
of MPI_Finalize(). As shown in this example, this can be used to make
sure that global MPI_Datatype variables associated to global datatypes
are freed by calling MPI_Type_free() before program exit (and thus
preventing ugly memory leaks/outstanding allocations when run in
valgrind for example). This mechanism is used by the library I'm working
on as well as by PetSc library.
The program works by taking advantage of the MPI-2 Standard
Section 4.8 "Allowing User Function at Process Termination". As it says,
the MPI_Finalize() function calls the delete callback associated to the
MPI_COMM_SELF attribute "before any other part of MPI are affected". It
also says that "calling MPI_Finalized() will return false in any of
these callback functions".
Section 4.9 of the MPI-2 Standard: "Determining Whether MPI Has
Finished" moreover says that it can be determined if MPI is active by
calling MPI_Finalized(). It also reaffirm that MPI is active in the
callback functions invoked by MPI_Finalize().
I think that an "active" MPI library here means that basic MPI
functions like MPI_Type_free() can be called.
The following small program therefore seems to conform to the
MPI standard.
However where I run it (compiled with OpenMPI 1.0.2 mpicc), I
get the following message:
*** An error occurred in MPI_Type_free
*** after MPI was finalized
*** MPI_ERRORS_ARE_FATAL (goodbye)
Note that this program works well with mpich2.
Please have a look at this problem.
Thanks,
Martin Audet
#include <assert.h>
#include <stddef.h>
#include <mpi.h>
static int attr_delete_function(MPI_Comm p_comm, int p_keyval,
void *p_attribute_val, void * p_extra_state)
{
assert(p_attribute_val != NULL);
/* Get a reference on the datatype received. */
MPI_Datatype *const cur_datatype = (MPI_Datatype
*)(p_attribute_val);
/* Free it if non null. */
if (*cur_datatype != MPI_DATATYPE_NULL) {
MPI_Type_free(cur_datatype);
assert(*cur_datatype == MPI_DATATYPE_NULL);
}
return MPI_SUCCESS;
}
/* If p_datatype refer to a non null MPI datatype, this function
will register a callback */
/* function to free p_datatype and set it to MPI_DATATYPE_NULL.
This callback will be */
/* called during the first steps of the MPI_Finalize() function
when the state of the MPI */
/* library still allows MPI functions to be called. This is
done by associating an */
/* attribute to the MPI_COMM_SELF communicator as allowed by
the MPI 2 standard (section 4.8). */
static void add_type_free_callback(MPI_Datatype *p_datatype)
{
int keyval;
assert(p_datatype != NULL);
/* First create the keyval.
*/
/* No callback function will be called when MPI_COMM_SELF is
duplicated */
/* and attr_delete_function() will be called when
MPI_COMM_SELF is */
/* freed (e.g. during MPI_Finalize()).
*/
/* Since many callback can be associated with MPI_COMM_SELF
to free many */
/* datatypes, a new keyval has to be created every time.
*/
MPI_Keyval_create(MPI_NULL_COPY_FN, &attr_delete_function,
&keyval, NULL);
/* Then associate this keyval to MPI_COMM_SELF and make sure
the pointer */
/* to the datatype p_datatype is passed to the callback.
*/
MPI_Attr_put(MPI_COMM_SELF, keyval, p_datatype);
/* Free the keyval because it is no longer needed.
*/
MPI_Keyval_free(&keyval);
}
typedef struct {
short ss;
int ii;
} glb_struct_t;
MPI_Datatype glb_dtype = MPI_DATATYPE_NULL;
static void calc_glb_dtype(void)
{
const int NB_MEM = 3;
static int len_tbl[3] = { 1,
1, 1 };
static MPI_Aint disp_tbl[3] = { offsetof(glb_struct_t,
ss), offsetof(glb_struct_t, ii), sizeof(glb_struct_t) };
static MPI_Datatype type_tbl[3] = { MPI_SHORT,
MPI_INT, MPI_UB };
MPI_Type_struct(NB_MEM, len_tbl, disp_tbl, type_tbl,
&glb_dtype);
MPI_Type_commit(&glb_dtype);
add_type_free_callback(&glb_dtype);
}
int main(int argc, char *argv[])
{
MPI_Init(&argc, &argv);
calc_glb_dtype();
MPI_Finalize();
return 0;
}
