Hello, Thanks for your description. I'm currently doing allToAll() prior to allToAllV(), to communicate length of expected messages. . BUT, I still strongly believe that the right implementation of this method is something that I expected earlier! If you check the MPI specification here:
http://www.mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf Page 170 Line 14 It is mentioned that "... the number of elements that CAN be received...". which implies that the actual received message may have shorter length. While in cases where it is mandatory to have same value, the modal "MUST" is used. for example at page 171 Line 1, it is mentioned that "... sendtype at process i MUST be equal to the type signature ...". SO, I would expect that any consistent implementation of MPI specification handle this message length matching by itself, as I asked originally. Thanks, -- HR On Tue, Apr 7, 2015 at 6:03 AM, Howard Pritchard <hpprit...@gmail.com> wrote: > Hi HR, > > Sorry for not noticing the receive side earlier, but as Ralph implied > earlier > in this thread, the MPI standard has more strict type matching for > collectives > than for point to point. Namely, the number of bytes the receiver expects > to receive from a given sender in the alltoallv must match the number of > bytes > sent by the sender. > > You were just getting lucky with the older open mpi. The error message > isn't so great though. Its likely in the newer open mpi you are using a > collective algorithm for alltoallv that assumes you're app is obeying the > standard. > > You are correct that if the ranks don't know how much data will be sent > to them from each rank prior to the alltoallv op, you will need to have > some > mechanism for exchanging this info prior to the alltoallv op. > > Howard > > > 2015-04-06 23:23 GMT-06:00 Hamidreza Anvari <hr.anv...@gmail.com>: > >> Hello, >> >> If I set the size2 values according to your suggestion, which is the same >> values as on sending nodes, it works fine. >> But by definition it does not need to be exactly the same as the length >> of sent data, and it is just a maximum length of expected data to receive. >> If not, it is inevitable to run a allToAll() first to communicate the data >> sizes, and then doing the main allToAllV(), which is an expensive >> unnecessary communication overhead. >> >> I just created a reproducer in C++ which gives the error under OpenMPI >> 1.8.4, but runs correctly under OpenMPI 1.5.4 . >> (I've not included the Java version of this reproducer, which I think is >> not important as current version is enough to reproduce the error. But in >> case, it is straight forward to convert this code to Java). >> >> Thanks, >> -- HR >> >> On Mon, Apr 6, 2015 at 3:03 PM, Ralph Castain <r...@open-mpi.org> wrote: >> >>> That would imply that the issue is in the underlying C implementation in >>> OMPI, not the Java bindings. The reproducer would definitely help pin it >>> down. >>> >>> If you change the size2 values to the ones we sent you, does the program >>> by chance work? >>> >>> >>> On Apr 6, 2015, at 1:44 PM, Hamidreza Anvari <hr.anv...@gmail.com> >>> wrote: >>> >>> I'll try that as well. >>> Meanwhile, I found that my c++ code is running fine on a machine running >>> OpenMPI 1.5.4, but I receive the same error under OpenMPI 1.8.4 for both >>> Java and C++. >>> >>> On Mon, Apr 6, 2015 at 2:21 PM, Howard Pritchard <hpprit...@gmail.com> >>> wrote: >>> >>>> Hello HR, >>>> >>>> Thanks! If you have Java 1.7 installed on your system would you mind >>>> trying to test against that version too? >>>> >>>> Thanks, >>>> >>>> Howard >>>> >>>> >>>> 2015-04-06 13:09 GMT-06:00 Hamidreza Anvari <hr.anv...@gmail.com>: >>>> >>>>> Hello, >>>>> >>>>> 1. I'm using Java/Javac version 1.8.0_20 under OS X 10.10.2. >>>>> >>>>> 2. I have used the following configuration for making OpenMPI: >>>>> ./configure --enable-mpi-java >>>>> --with-jdk-bindir="/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands" >>>>> --with-jdk-headers="/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers" >>>>> --prefix="/users/hamidreza/openmpi-1.8.4" >>>>> >>>>> make all install >>>>> >>>>> 3. As a logical point of view, size2 is the maximum expected data to >>>>> receive, which in turn might be less that this maximum. >>>>> >>>>> 4. I will try to prepare a working reproducer of my error and send it >>>>> to you. >>>>> >>>>> Thanks, >>>>> -- HR >>>>> >>>>> On Mon, Apr 6, 2015 at 10:46 AM, Ralph Castain <r...@open-mpi.org> >>>>> wrote: >>>>> >>>>>> I've talked to the folks who wrote the Java bindings. One possibility >>>>>> we identified is that there may be an error in your code when you did the >>>>>> translation >>>>>> >>>>>> My immediate thought is that each process can not receive more >>>>>> elements than it was sent to them. That's the reason of truncation error. >>>>>> >>>>>> These are the correct values: >>>>>> >>>>>> rank 0 - size2: 2,2,1,1 >>>>>> rank 1 - size2: 1,1,1,1 >>>>>> rank 2 - size2: 0,1,1,2 >>>>>> rank 3 - size2: 2,1,2,1 >>>>>> >>>>>> >>>>>> Can you check your code to see if perhaps the values you are passing >>>>>> didn't get translated correctly from your C++ version to the Java >>>>>> version? >>>>>> >>>>>> >>>>>> >>>>>> On Apr 6, 2015, at 5:03 AM, Howard Pritchard <hpprit...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Hello HR, >>>>>> >>>>>> It would also be useful to know which java version you are using, as >>>>>> well >>>>>> as the configure options used when building open mpi. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Howard >>>>>> >>>>>> >>>>>> >>>>>> 2015-04-05 19:10 GMT-06:00 Ralph Castain <r...@open-mpi.org>: >>>>>> >>>>>>> If not too much trouble, can you extract just the alltoallv portion >>>>>>> and provide us with a small reproducer? >>>>>>> >>>>>>> >>>>>>> On Apr 5, 2015, at 12:11 PM, Hamidreza Anvari <hr.anv...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am converting an existing MPI program in C++ to Java using OpenMPI >>>>>>> 1.8.4, >>>>>>> At some point I have a allToAllv() code which works fine in C++ but >>>>>>> receives error in Java version: >>>>>>> >>>>>>> MPI.COMM_WORLD.allToAllv(data, subpartition_size, >>>>>>> subpartition_offset, MPI.INT <http://mpi.int/>, >>>>>>> data2,subpartition_size2,subpartition_offset2,MPI.INT >>>>>>> <http://mpi.int/>); >>>>>>> >>>>>>> Error: >>>>>>> *** An error occurred in MPI_Alltoallv >>>>>>> *** reported by process [3621322753,9223372036854775811] >>>>>>> *** on communicator MPI_COMM_WORLD >>>>>>> *** MPI_ERR_TRUNCATE: message truncated >>>>>>> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now >>>>>>> abort, >>>>>>> *** and potentially your MPI job) >>>>>>> 3 more processes have sent help message help-mpi-errors.txt / >>>>>>> mpi_errors_are_fatal >>>>>>> Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / >>>>>>> error messages >>>>>>> >>>>>>> Here are the values for parameters: >>>>>>> >>>>>>> data.length = 5 >>>>>>> data2.length = 20 >>>>>>> >>>>>>> ---------- Rank 0 of 4 ---------- >>>>>>> subpartition_offset:0,2,3,3, >>>>>>> subpartition_size:2,1,0,2, >>>>>>> subpartition_offset2:0,5,10,15, >>>>>>> subpartition_size2:5,5,5,5, >>>>>>> ---------- >>>>>>> ---------- Rank 1 of 4 ---------- >>>>>>> subpartition_offset:0,2,3,4, >>>>>>> subpartition_size:2,1,1,1, >>>>>>> subpartition_offset2:0,5,10,15, >>>>>>> subpartition_size2:5,5,5,5, >>>>>>> ---------- >>>>>>> ---------- Rank 2 of 4 ---------- >>>>>>> subpartition_offset:0,1,2,3, >>>>>>> subpartition_size:1,1,1,2, >>>>>>> subpartition_offset2:0,5,10,15, >>>>>>> subpartition_size2:5,5,5,5, >>>>>>> ---------- >>>>>>> ---------- Rank 3 of 4 ---------- >>>>>>> subpartition_offset:0,1,2,4, >>>>>>> subpartition_size:1,1,2,1, >>>>>>> subpartition_offset2:0,5,10,15, >>>>>>> subpartition_size2:5,5,5,5, >>>>>>> ---------- >>>>>>> >>>>>>> Again, this is a code which works in C++ version. >>>>>>> >>>>>>> Any help or advice is greatly appreciated. >>>>>>> >>>>>>> Thanks, >>>>>>> -- HR >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26610.php >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26613.php >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26615.php >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26616.php >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2015/04/26617.php >>>>> >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2015/04/26620.php >>>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2015/04/26622.php >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2015/04/26623.php >>> >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/04/26631.php >> > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/04/26634.php >
