Gus,
Thanks for the pointer to the subarray and darray bug in MPICH. It turns out
that Open MPI has a similar issue. I guess we both followed the MPI standard
literally, and the standard specifically mention the LB and UB marker in the
construction of these two types. I pushed a fix for Open MPI, that will
eventually make it’s way toward the next stable release.
Thanks,
George.
> On Jan 16, 2015, at 17:23 , Gus Correa <g...@ldeo.columbia.edu> wrote:
>
> Hi George
>
> Many thanks for your answer and interest in my questions.
> ... so ... more questions inline ...
>
> On 01/16/2015 03:41 PM, George Bosilca wrote:
>> Gus,
>>
>> Please see my answers inline.
>>
>>> On Jan 16, 2015, at 14:24 , Gus Correa <g...@ldeo.columbia.edu> wrote:
>>>
>>> Hi George
>>>
>>> It is still not clear to me how to deal with strides in
>>> MPI_Create_type_subarray.
>>> The function/subroutine interface doesn’t mention strides at all.
>>
>> That’s indeed a little tricky.
>> However, the trick here is that when you try to understand the subarray
>> type you should think recursively about the datatypes involved in the
>> operation.
>>
>>> It is a pity that there is little literature (books) about MPI,
>>> and the existing books are lagging behind the new MPI developments and
>>> standards (MPI-2, MPI-3).
>>> My most reliable sources so far were the "MPI - The complete reference"
>>> books, vol-1 (2nd ed.) and vol-2 (which presumably covers MPI-2).
>>> However, they do not even mention MPI_Create_type_subarray,
>>> which is part of the MPI-2 standard.
>>
>> Let me do a wild guess: you two guys must be the firsts to ask questions
>> about it …
>>
>
> Did anybody but the MPI developers actually *used* MPI_Create_type_subarray?
> Could this explain the scarcity of questions about it? :)
>
>>> I found it in the MPI-2 standard on the web, but it is not clear to me
>>> how to achieve the same effect of strides that are available in
>>> MPI_Type_vector.
>>> MPI_Create_type_subarray is in section 4.1.3.
>>> The OMPI MPI_Create_type_subarray man page says there is an example in
>>> section 9.9.2 of the MPI-2 standard.
>>> However, there is no section 9.9.2.
>>> Chapter 9 is about something else ("The info object"), not derived types.
>>> No good example of MPI_Create_type_subarray in section 4.1.3 either,
>>> which is written in the typical terse and hermetic style in which
>>> standards are.
>>
>> No comments on subjective topics … ;)
>
> It flows almost as smoothly as Backus-Naur prose.
> Makes a great reading with some Mozart in the background.
>
>> You just blew my day away, I was totally under the impression that the
>> MPI standard reads like a children’s bedtime story book !!!
>>
>
>
> Did you write it?
> Do you read it for your kids at bed time?
> Do they fall asleep right away?
>
> Oh, if AEsop, Grimm Brothers, Charles Perrault, Andersen could only have
> helped as copy-desks ...
>
>
>>> So, how can one handle strides in MPI_Create_type_subarray?
>>> Would one have to first create several MPI_Type_vector for the various
>>> dimensions, then use them as "oldtype" in MPI_Create_type_subarray?
>>> That sounds awkward, because there is only one “oldtype" in
>>> MPI_Create_type_subarray, not one for each dimension.
>>
>> Exactly. Take a look at how we handle the subarray in Open MPI,
>> more precisely at the file ompi/datatype/ompi_datatype_create_subarray.c.
>> My comment from few days ago referred exactly to this code, where the
>> subarray is basically described in terms of vector
>> (well maybe vector as I was lazy to check the LB/UB).
>>
>
> When documentation equates to reading actual function code ...
> ... that is when users drop trying to use new developments ...
>
> BTW, ominously a bug report on LB/UB misuse in MPI_Type_struct
> *and* in MPI_Type_create_subarray ... gosh ...
>
> http://lists.mpich.org/pipermail/discuss/2015-January/003608.html
>
> But hopefully that doesn't affect Open MPI, right?
>
>> As I said above think recursively.
>> You start with the old type,
>> then build another try on a dimension,
>> then you use this to expose the second dimensions and so on.
>> For each dimension your basic type is not the user provided old type,
>> but the type you built so far.
>>
>> - size_array[i] is the number of elements in big data in the dimension i
>> - subsize_array[i] is the of element you will include in your datatype in
>> the dimension i
>> - start_array[i] is how many elements you will skip in the dimension i
>> before you start including them in your datatype. start[i] + subside[i] must
>> be smaller or equal to size[i]
>>
>
> OK, that starts to make more sense than the yawning bedtime story
> in the MPI-2 standard.
>
> I should peel off (that would be recursive)
> or build up (that would be non-recursive, right?) each dimension,
> one by one,
> like an onion,
> creating new subarrays of increasing dimensions,
> one by one,
> based on the subarray previously created.
> Did I get it right?
> So, should I peel off or build up the dimensions?
>
> In which regard is this any better than using MPI_Type_Vector,
> which can be setup in a single non-recursive call,
> as long as the sizes, strides, etc, are properly calculated?
>
>>> Is there any simple example of how to achieve stride effect with
>>> MPI_Create_type_subarray in a multi-dimensional array?
>>
>> Not as far as I know.
>> But now that people expressed interest in this topic,
>> maybe someone will write a blog or something about.
>>
>
> An example, just a simple example ...
> ... to help those that have to write all steps from 1 to N,
> when it comes to thinking recursively ...
> When it comes to recursion, I stopped at the Fibonacci numbers.
>
> Well, even if it is on a blog ...
> Nobody seem to care about books or printed matter anymore.
> It is all about blogs, wikis, ..., please don't move the OMPI documentation
> to Twitter, Facebook, or Instagram OK?
> I don't have an account on any of those.
> At least keep the FAQ on the web.
> Our local library here was phased out,
> will be replaced with a media center ...
> When I hear news like those I always think of
> Ray Bradbury's Fahrenheit 451 (and Franc,ois Truffaut's beautiful movie).
>
>>> BTW, when are you gentlemen going to write an updated version of the
>>> “MPI - The Complete Reference"? :)
>>
>> Maybe after the release of MPI 4.0 would be a good target …
>
> Not any sooner than that?
> MPI-2 is already poorly covered in the literature,
> MPI-3 only by the standard (yawn ...).
> And when MPI 4 comes, would we have to wait for MPI-5 to get
> the examples?
>
>> A lot of new and exciting technologies will hopefully be going in by then,
>
> ... which won't be used if people don't get the word about them,
> on how to use them effectively ... please, don't let MPI-3-4-5-6-
> become a [MPI-]chat only amongst the [[[>>>MPI developers<<<]]] ...
>
>> writing a new book might be worth the effort.
>>
>
> Always worth it, with examples, etc.
>
>> George.
>>
>
> Thank you, George!
>
> Gus Correa
>>
>>
>>
>>
>>>
>>> Thank you,
>>> Gus Correa
>>> (Hijacking Diego Avesani's thread, apologies to Diego.)
>>> (Also, I know this question is not about Open MPI, but about MPI in
>>> general. But the lack of examples may warrant asking the question here.)
>>>
>>>
>>> On 01/16/2015 01:39 AM, George Bosilca wrote:
>>>> The subarray creation is an multi-dimension extension of the vector type.
>>> You can see it as a vector of vector of vector and so on, one vector per
>>> dimension.
>>> The stride array is used to declare on each dimension what is the relative
>>> displacement
>>> (in number of elements) from the beginning of the dimension array.
>>>>
>>>> It is important to use regular type creation when you can take advantage
>>>> of such
>>> regularity instead of resorting to use of struct or h*. This insure better
>>> packing/unpacking performance, as well as possible future support for
>>> one-sided
>>> communications.
>>>>
>>>> George.
>>>>
>>>>
>>>>
>>>>> On Jan 15, 2015, at 19:31, Gus Correa <g...@ldeo.columbia.edu> wrote:
>>>>>
>>>>> I never used MPI_Type_create_subarray, only MPI_Type_Vector.
>>>>> What I like about MPI_Type_Vector is that you can define a stride,
>>>>> hence you can address any regular pattern in memory.
>>>>> However, it envisages the array layout in memory as a big 1-D array,
>>>>> with a linear index progressing in either Fortran or C order.
>>>>>
>>>>> Somebody correct me please if I am wrong, but at first sight
>>>>> MPI_Type_Vector sounds more flexible to me than MPI_Type_create_subarray,
>>>>> exactly because the latter doesn't have strides.
>>>>>
>>>>> The downside is that you need to do some index arithmetic to figure
>>>>> the right strides, etc, to match the corresponding
>>>>> Fortran90 array sections.
>>>>>
>>>>> There are good examples in the "MPI - The complete reference" books I
>>>>> suggested to you before (actually in vol 1).
>>>>>
>>>>> Online I could find the two man pages (good information, but no example):
>>>>>
>>>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_vector.3.php
>>>>> http://www.open-mpi.org/doc/v1.8/man3/MPI_Type_create_subarray.3.php
>>>>>
>>>>> There is a very simple 2D example of MPI_Type_vector using strides here:
>>>>>
>>>>> https://computing.llnl.gov/tutorials/mpi/#Derived_Data_Types
>>>>>
>>>>> and a similar one here:
>>>>>
>>>>> http://static.msi.umn.edu/tutorial/scicomp/general/MPI/content6.html
>>>>>
>>>>> Gus Correa
>>>>>
>>>>>> On 01/15/2015 06:53 PM, Diego Avesani wrote:
>>>>>> dear George, dear Gus, dear all,
>>>>>> Could you please tell me where I can find a good example?
>>>>>> I am sorry but I can not understand the 3D array.
>>>>>>
>>>>>>
>>>>>> Really Thanks
>>>>>>
>>>>>> Diego
>>>>>>
>>>>>>
>>>>>> On 15 January 2015 at 20:13, George Bosilca <bosi...@icl.utk.edu
>>>>>> <mailto:bosi...@icl.utk.edu>> wrote:
>>>>>>
>>>>>>
>>>>>>> On Jan 15, 2015, at 06:02 , Diego Avesani <diego.aves...@gmail.com
>>>>>>> <mailto:diego.aves...@gmail.com>> wrote:
>>>>>>>
>>>>>>> Dear Gus, Dear all,
>>>>>>> Thanks a lot.
>>>>>>> MPI_Type_Struct works well for the first part of my problem, so I
>>>>>>> am very happy to be able to use it.
>>>>>>>
>>>>>>> Regarding MPI_TYPE_VECTOR.
>>>>>>>
>>>>>>> I have studied it and for simple case it is clear to me what id
>>>>>>> does (at least I believe). Foe example if I have a matrix define as:
>>>>>>> REAL, ALLOCATABLE (AA(:,:))
>>>>>>> ALLOCATE AA(100,5)
>>>>>>>
>>>>>>> I could send part of it defining
>>>>>>>
>>>>>>> CALL MPI_TYPE_VECTOR(5,1,5,MPI_DOUBLE_PRECISION,/MY_NEW_TYPE/)
>>>>>>>
>>>>>>> after that I can send part of it with
>>>>>>>
>>>>>>> CALL MPI_SEND( AA(1:/10/,:), /10/, /MY_NEW_TYPE/, 1, 0,
>>>>>>> MPI_COMM_WORLD );
>>>>>>>
>>>>>>> Have I understood correctly?
>>>>>>>
>>>>>>> What I can do in case of three dimensional array? for example
>>>>>>> AA(:,:,:), I am looking to MPI_TYPE_CREATE_SUBARRAY.
>>>>>>> Is that the correct way?
>>>>>>>
>>>>>>> Thanks again
>>>>>>
>>>>>> Indeed, using the subarray is the right approach independent on the
>>>>>> number of dimensions of the data (you can use it instead of
>>>>>> MPI_TYPE_VECTOR as well).
>>>>>>
>>>>>> George.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Diego
>>>>>>>
>>>>>>>
>>>>>>> On 13 January 2015 at 19:04, Gus Correa <g...@ldeo.columbia.edu
>>>>>>> <mailto:g...@ldeo.columbia.edu>> wrote:
>>>>>>>
>>>>>>> Hi Diego
>>>>>>> I guess MPI_Type_Vector is the natural way to send and receive
>>>>>>> Fortran90 array sections (e.g. your QQMLS(:,50:100,:)).
>>>>>>> I used that before and it works just fine.
>>>>>>> I think that is pretty standard MPI programming style.
>>>>>>> I guess MPI_Type_Struct tries to emulate Fortran90 and C
>>>>>>> structures
>>>>>>> (as you did in your previous code, with all the surprises
>>>>>>> regarding alignment, etc), not array sections.
>>>>>>> Also, MPI type vector should be more easy going (and probably
>>>>>>> more efficient) than MPI type struct, with less memory
>>>>>>> alignment problems.
>>>>>>> I hope this helps,
>>>>>>> Gus Correa
>>>>>>>
>>>>>>> PS - These books have a quite complete description and several
>>>>>>> examples
>>>>>>> of all MPI objects and functions, including MPI types (native
>>>>>>> and user defined):
>>>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-0
>>>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-0>
>>>>>>> http://mitpress.mit.edu/books/__mpi-complete-reference-1
>>>>>>> <http://mitpress.mit.edu/books/mpi-complete-reference-1>
>>>>>>>
>>>>>>> [They cover MPI 1 and 2. I guess there is a new/upcoming book
>>>>>>> with MPI 3, but for what you're doing 1 and 2 are more than
>>>>>>> enough.]
>>>>>>>
>>>>>>>
>>>>>>> On 01/13/2015 09:22 AM, Diego Avesani wrote:
>>>>>>>
>>>>>>> Dear all,
>>>>>>>
>>>>>>> I had some wonderful talking about MPI_type_create_struct adn
>>>>>>> isend\irecv with
>>>>>>> Gilles, Gustavo, George, Gus, Tom and Jeff. Now all is
>>>>>>> more clear and my
>>>>>>> program works.
>>>>>>>
>>>>>>> Now I have another question. In may program I have matrix:
>>>>>>>
>>>>>>> /QQMLS(:,:,:) /that is allocate as
>>>>>>>
>>>>>>> /ALLOCATE(QQMLS(9,npt,18)/), where npt is the number of
>>>>>>> particles
>>>>>>>
>>>>>>> QQMLS is double precision.
>>>>>>>
>>>>>>> I would like to sent form a CPU to another part of it, for
>>>>>>> example,
>>>>>>> sending QQMLS(:,50:100,:). I mean sending the QQMLS of the
>>>>>>> particles
>>>>>>> between 50 to 100.
>>>>>>> I suppose that i could use MPI_Type_vector but I am not
>>>>>>> sure. The
>>>>>>> particle that I want to sent could be from 25 to 50 ecc..
>>>>>>> ecc..so
>>>>>>> blocklength changes everytime.
>>>>>>>
>>>>>>> Do I have to use MPI_type_create_struct?
>>>>>>> Do I have correctly understood MPI_Type_vector?
>>>>>>>
>>>>>>> Thanks a lot
>>>>>>>
>>>>>>>
>>>>>>> Diego
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>> Subscription:
>>>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users
>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>>>>>> Link to this post:
>>>>>>>
>>>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/__26171.php
>>>>>>>
>>>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/26171.php>
>>>>>>>
>>>>>>>
>>>>>>> _________________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>> Subscription:
>>>>>>> http://www.open-mpi.org/__mailman/listinfo.cgi/users
>>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>>>>>> Link to this post:
>>>>>>>
>>>>>>> http://www.open-mpi.org/__community/lists/users/2015/01/__26172.php
>>>>>>> <http://www.open-mpi.org/community/lists/users/2015/01/26172.php>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>> Link to this post:
>>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26184.php
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26192.php
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>> Link to this post:
>>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26193.php
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>> Link to this post:
>>>>> http://www.open-mpi.org/community/lists/users/2015/01/26194.php
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>> Link to this post:
>>>> http://www.open-mpi.org/community/lists/users/2015/01/26195.php
>>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2015/01/26206.php
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2015/01/26208.php
>>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2015/01/26210.php
