Bogdan,
The bug was solely related to the number of entries in the datatype, and
not the number of elements nor the size/extent of the datatype. As such, 64
bits support was not impacted by this bug.
>From the user perspective, the only visible improvement is the possibility
to create datatypes with more than 10k (or so) different components. Thus,
it might only affect datatype creation routines that provides arrays as
arguments (mainly indexed and struct).
George.
On Fri, Mar 6, 2015 at 1:28 PM, Bogdan Sataric <bogdan.sata...@gmail.com>
wrote:
> George thank you very much!
>
> So can I assume that new indexed type in 1.8.5 will support 64-bit large
> datatypes, or over the current 4GB datatypes (and some strange internal
> restrictions in my case)? Or is there any clue what will be the improvement
> over the existing datatype restrictions?
>
> Regards,
>
> ----
>
> Bogdan Sataric
>
> email: bogdan.sata...@gmail.com
> phone: +381 21-485-2441
>
> Teaching & Research Assistant
> Chair for Applied Computer Science
> Faculty of Technical Sciences, Novi Sad, Serbia
>
> On Fri, Mar 6, 2015 at 12:52 AM, George Bosilca <bosi...@icl.utk.edu>
> wrote:
>
>> On Thu, Mar 5, 2015 at 6:22 PM, Bogdan Sataric <bogdan.sata...@gmail.com>
>> wrote:
>>
>>> Hello George,
>>>
>>> So is it safe for me to assume that my code is good and that you will
>>> remove this bug from next OpenMPI version?
>>>
>>
>> Yes I think it is safe to assume your code is correct (or at least it
>> follows the specifications you describe in your email).
>>
>> Also I would like to know which future OpenMPI version will incorporate
>>> this fix (so I can try my code in fixed version)?
>>>
>>
>> I pushed the code in the trunk, and created a request to get it into the
>> 1.8.5. So you can try any nightly build starting from tonight, and then any
>> stable release after the 1.8.4
>>
>> George.
>>
>>
>>>
>>
>>>
>>> Thank you,
>>>
>>> ----
>>>
>>> Bogdan Sataric
>>>
>>> email: bogdan.sata...@gmail.com
>>> phone: +381 21-485-2441
>>>
>>> Teaching & Research Assistant
>>> Chair for Applied Computer Science
>>> Faculty of Technical Sciences, Novi Sad, Serbia
>>>
>>> On Thu, Mar 5, 2015 at 6:31 PM, George Bosilca <bosi...@icl.utk.edu>
>>> wrote:
>>>
>>>> Bogdan,
>>>>
>>>> As far as I can tell your code is correct, and the problem is coming
>>>> from Open MPI. More specifically, I used alloca in the optimization stage
>>>> in MPI_Type_commit, and as your arrays of length were too large, alloca
>>>> failed and lead to a segfault. I fixed in the trunk (3c489ea), and this
>>>> will get into our next release.
>>>>
>>>> Unfortunately there is no fix for the 1.6 that I can think of.
>>>> Apparently, you are really the first that run into such kind of problems,
>>>> so guess you are the first creating gigantic datatypes.
>>>>
>>>> Thanks for the bug report,
>>>> George.
>>>>
>>>>
>>>> On Thu, Mar 5, 2015 at 9:09 AM, Bogdan Sataric <
>>>> bogdan.sata...@gmail.com> wrote:
>>>>
>>>>> I've been having problems with my 3D matrix transpose program. I'm
>>>>> using MPI_Type_indexed in order to allign specific blocks that I want to
>>>>> send and receive across one or multiple nodes of a cluster. Up to few days
>>>>> ago I was able to run my program without any errors. However several test
>>>>> cases on the cluster in last few days exposed segmentation fault when I
>>>>> try
>>>>> to form indexed type on some specific matrix configurations.
>>>>>
>>>>> The code that forms indexed type is as follows:
>>>>>
>>>>> #include <stdio.h>
>>>>> #include <stdlib.h>
>>>>> #include <mpi.h>
>>>>>
>>>>> int main(int argc, char **argv) {
>>>>>
>>>>> int Nx = 800;
>>>>> int Ny = 640;
>>>>> int Nz = 480;
>>>>> int gsize;
>>>>> int i, j;
>>>>>
>>>>> MPI_Init(&argc, &argv);
>>>>> MPI_Comm_size(MPI_COMM_WORLD, &gsize);
>>>>>
>>>>> printf("GSIZE: %d\n", gsize);
>>>>>
>>>>> MPI_Datatype double_complex_type;
>>>>> MPI_Datatype block_send_complex_type;
>>>>>
>>>>> int * send_displ = (int *) malloc(Nx * Ny/gsize * sizeof(int));
>>>>> int * send_blocklen = (int *) malloc(Nx * Ny/gsize * sizeof(int));
>>>>>
>>>>> MPI_Type_contiguous(2, MPI_DOUBLE, &double_complex_type);
>>>>> MPI_Type_commit(&double_complex_type);
>>>>>
>>>>> for (i = Ny/gsize - 1; i >= 0 ; i--) {
>>>>> for (j = 0; j < Nx; j++) {
>>>>> send_displ[(Ny/gsize - 1 - i) * Nx + j] = i * Nz + j * Ny
>>>>> * Nz;
>>>>> send_blocklen[(Ny/gsize - 1 - i) * Nx + j] = Nz;
>>>>> }
>>>>> }
>>>>>
>>>>> MPI_Type_indexed(Nx * Ny/gsize, send_blocklen, send_displ,
>>>>> double_complex_type, &block_send_complex_type);
>>>>> MPI_Type_commit(&block_send_complex_type);
>>>>>
>>>>> free(send_displ);
>>>>> free(send_blocklen);
>>>>>
>>>>> MPI_Finalize();
>>>>> }
>>>>>
>>>>> Values of the Nx, Ny and Nz respectively are 800, 640 and 480. Value
>>>>> of gsize for this test was 1 (simulation of MPI program on 1 node). The
>>>>> node has 32GB of RAM and no other memory has been allocated (only this
>>>>> code
>>>>> has been run).
>>>>>
>>>>> In code basically I'm creating double_complex_type to represent
>>>>> complex number (2 contiguous MPI_DOUBLE) values. The whole matrix has 800
>>>>> *
>>>>> 640 * 480 of these values and I'm trying to catch these values in the
>>>>> indexed type. One indexed type block length is the whole Nz "rod" while
>>>>> ordering of these "rods" in displacements array is given by the formula i
>>>>> *
>>>>> Nz + j * Ny * Nz. Basically displacements start from top row, and left
>>>>> column of the 3D matrix. Then I gradually sweep to the right sight of that
>>>>> top row, then go to one row below sweep to the right side and so on until
>>>>> the bottom row.
>>>>>
>>>>> The strange thing is that this formula and algorithm *WORK* if I put
>>>>> MPI_DOUBLE type instead of derived complex type (1 instead of 2 in
>>>>> MPI_TYPE_CONTIGIOUS). Also this formula *WORKS* if I put 1 for Nz
>>>>> dimension instead of 480. However if I change Nz to even 2 I get
>>>>> segmentation fault error in the MPI_Type_commit call.
>>>>>
>>>>> I checked all of the displacements and they seem fine. There is no
>>>>> overlapping of displacements or going under 0 or over extent of the formed
>>>>> indexed type. Also the size of the datatype is below 4GB (which is I
>>>>> believe limit of MPI datatypes (since MPI_Type_size function returns int *
>>>>> ). Also I believe amount of memory is not an issue as even if I put Nz to
>>>>> be 2, I get the same segmentation fault error, and the node has 32GB of
>>>>> RAM
>>>>> just for this test.
>>>>>
>>>>> What bothers me is that most of other indexed type configurations
>>>>> (with plain MPI_DOUBLE type elements), or complex type with smaller matrix
>>>>> (say 400 * 640 * 480) *WORK* without segmentation fault. Also If I
>>>>> commit the indexed type with MPI_DOUBLE type elements even larger matrices
>>>>> work (say 960 x 800 x 640) which have exactly the same type size as 800 x
>>>>> 640 x 480 complex indexed type (just under 4GB)! So basically the type
>>>>> size
>>>>> is not an issue here, but somehow either number of blocks, size of
>>>>> particular blocks, or size of block elements create problems. I'm not sure
>>>>> weather there is problem in implementation of OpenMPI or something in my
>>>>> code is wrong...
>>>>>
>>>>> I would greatly appreciate any help as I've been stuck on this problem
>>>>> for days now and nothing in MPI documentation and the examples I found on
>>>>> the internet is giving me a clue where the error might be.
>>>>>
>>>>> At the end I would like to say that code has been compiled with
>>>>> Open-MPI version 1.6.5.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Bogdan Sataric
>>>>> ----
>>>>>
>>>>> Bogdan Sataric
>>>>>
>>>>> email: bogdan.sata...@gmail.com
>>>>> phone: +381 21-485-2441
>>>>>
>>>>> Teaching & Research Assistant
>>>>> Chair for Applied Computer Science
>>>>> Faculty of Technical Sciences, Novi Sad, Serbia
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/users/2015/03/26430.php
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/03/26431.php
>>>>
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2015/03/26433.php
>>>
>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/03/26435.php
>>
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/03/26440.php
>
