Hi George, Thank you for the prompt reply. Indeed we are constructing a data-type description with more than 32k entries.
I attached a screenshot of the pData structure (displayed with the TotalView debugger), I hope this helps you. Unfortunately I was not able to use gdb to execute the call you mentioned. Let me explain the relation of our code with the BOOST libraries: The code I'm debugging at the moment does not use any BOOST library to interface MPI, but it uses the same ideas of how to automatically create the data-types as the BOOST Parallel/Message Passing/MPI [1] library. This is due to the fact that the library is based on our ideas and the goal to factor out our message passing code into an open-source library (see [2]). Even though such an automatically created data-type description might not lead to an optimal performance, I think large descriptors should be supported for several reasons: - even when using MPI, not all parts of the code are performance critical - other MPI implementations support it - passing large/complicated data structures with the BOOST Parallel/Message Passing/MPI library (which supports LAM, MPICH and Open-MPI out of the box, see [3]) will probably lead to the same effect. - the fix has minor to no impact on rest of the code base, at least an appropriate error handling would be expected in the case of a too large data type descriptor. I hope that we are now sure that we have identified the problem as well as the solution and that you are willing to fix the issue in upcoming releases of Open-MPI. If there is anything else I can help with, please let me know. Regards, Michael Gauckler [1] http://lists.boost.org/Archives/boost/2007/01/115347.php [2] http://lists.boost.org/boost-announce/2006/09/0099.php [3] http://www.generic-programming.org/~dgregor/boost.mpi/boost/parallel/mpi/ -----Original Message----- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of George Bosilca Sent: Thursday, April 19, 2007 12:15 AM To: Open MPI Users Subject: Re: [OMPI users] Datatype construction, serious limitation (was: Signal: Segmentation fault (11) Problem) I am the developer and the maintainer of the data-type engine in Open MPI. And, I'm stunned (!) It never occur to me that someone will ever use a data-type description that need more than 32K entries on the internal stack. Let me explain a little bit. The stack is used to efficiently parse the data-type description. The 32K limit it's not a limit for the number of predefined MPI types in the data-type, but a limit for the number of different data descriptions (a description is like a vector of a predefined type). As an example an MPI_Type_struct with count 10 will use 11 entries. So in order to overload this data description one has to use an MPI_Type_struct with a count bigger than 32K (which might be the case with the BOOST library you're using in your code). In conclusion if your data-type description contain more than 32K entries, the current implementation will definitively not work for you. How many entries are in your data-type description ? There is an easy way to figure out if this is the problem with your code. Attaching gdb to your process and setting a break in the ompi_generic_simple_pack function is the first step. Once there, doing in gdb "call ompi_ddt_dump(pData)" will print a high level description of the data as represented internally in Open MPI. If you can provide the output of this call I can tell you in few seconds if this is the real issue or not. However, this raise another question about the performance you expect from your code. A data description with more than 32K items, cannot be efficiently optimized by any automatic data-type engine. Moreover, it cannot be easily parsed. I suggest that if it's possible to identify access patterns that are repetitive, one should use them in order to improve the data-type description. Thanks, george. On Apr 18, 2007, at 4:16 PM, Michael Gauckler wrote: > Dear Open-MPI Developers, > > investigations on the segmentation fault (see previous postings > "Signal: Segmentation fault (11) Problem") lets us suspect that > Open-MPI allows only a limited number of elements in the description > of user-defined MPI_Datatypes. > > Our application segmentation-faults when a large user-defined data > structure is passed to MPI_Send. > > The segmentation fault happens in the function > ompi_generic_simple_pack in datatype_pack.c when trying to access > pElem (Bad address). The structure pElem is set in line 276, where it > is retrieved as > > 276: pElem = &(description[pos_desc]); > > pos_desc is of type uint32_t with the value 0xffff929f (4294939295), > which itself is set on line 271 by a variable of type int16_t and > value -1. This leads to the indexing of the description structure at > position -1, producing the segmentation fault. The origin of the > pos_desc can be faund in the same function at line 271: > > 271: pos_desc = pStack->index; > > The structure to which pStack is pointing is of type dt_stack, defined > in ompi/datatype/convertor.h starting at line 65, where index is and > int16_t and commented with "index in the element > description": > > > typedef struct dt_stack { > > int16_t index; /**< index in the element description */ > > int16_t type; /**< the type used for the last pack/unpack > (original or DT_BYTE) */ > > size_t count; /**< number of times we still have to do it */ > > ptrdiff_t disp; /**< actual displacement depending on the > count field */ > > } dt_stack_t; > > > We therefore conclude that MPI_Datatypes, which are constructed with > Open-MPI (in the release of 1.2.1a of April 10th 2007) have the > limitation of containing a maximum of 32'768 separate entries. > > Although changing the type of the index to int32_t solves the problem > of the segmentation fault, I would be happy if the author / maintainer > of the code could have a look at it and decide if this is viable fix. > Having spent a lot of time in hunting down the issue into the Open-MPI > code, I would be glad to see the issue fixed in upcoming releases. > > Thanx and regards, > Michael Gauckler > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
