I totally agree. Portability is the most important thing for me. That's why the virtual router is not required to run the software.
On 28/09/12 10:32 AM, Iliev, Hristo wrote: >>From my understanding of the Open MPI code, it uses negative tags to >>implement collective operations and one-sided communication only. Your code >>might work if you use neither of them but will not be portable, not only >>across MPI implementations, but also some future change in Open MPI might >>break it. > > Just my €0.02. > > Kind regards, > Hristo Iliev > -- > Hristo Iliev, Ph.D. -- High Performance Computing > RWTH Aachen University, Center for Computing and Communication > Rechen- und Kommunikationszentrum der RWTH Aachen > Seffenter Weg 23, D 52074 Aachen (Germany) > > >> -----Original Message----- >> From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] >> On Behalf Of Sébastien Boisvert >> Sent: Friday, September 28, 2012 3:09 PM >> To: us...@open-mpi.org >> Subject: Re: [OMPI users] About MPI_TAG_UB >> >> Hello, >> >> My application has 191 MPI tags allocated with allocateMessageTagHandle, >> so >> 7 bits is not enough. >> >> Indeed, tags can be valued from 0 to 2147483647 inclusively with this >> MPI_TAG_UB value in Open-MPI. I misused the returned pointer. >> >> In Open-MPI, MPI_ANY_TAG is -1. I removed the boundary check in >> MPI_Isend and MPI_Recv to allow values from -2147483648 to 2147483647 >> inclusively. >> >> https://raw.github.com/sebhtml/patches/master/ompi-1.6.2-ray-4096- >> routing.patch >> >> As long as my tag is not MPI_ANY_TAG, I guess it should work fine although it >> is not MPI-compliant. I will test that. >> >> On 28/09/12 03:50 AM, Iliev, Hristo wrote: >>> Hello, >>> >>> MPI_TAG_UB in Open MPI is INT_MAX == 2^31-1 == 2147483647. The value >>> of >>> 17438272 (0x10A1640) is a bit strange for MPI_TAG_UB. I would rather >>> say that it is the value of a pointer to someplace in the heap, i.e. >>> you have missed the fact that the attribute value as returned by >>> MPI_Comm_get_attr / MPI_Attr_get is a pointer to the actual value (for >>> MPI_TAG_UB it is a pointer to int). >>> >>> MPI_TAG_UB is predefined attribute and according to §8.1.2 of the MPI >>> standard its value cannot be changed by the user. >>> >>> You have to find another solution, e.g. reduce the tag space to 7 bits >>> or put the routing info inside the message payload. >>> >>> Best regards, >>> Hristo Iliev >>> -- >>> Hristo Iliev, Ph.D. -- High Performance Computing RWTH Aachen >>> University, Center for Computing and Communication >>> Rechen- und Kommunikationszentrum der RWTH Aachen Seffenter Weg >> 23, D >>> 52074 Aachen (Germany) >>> >>>> -----Original Message----- >>>> From: users-boun...@open-mpi.org [mailto:users-bounces@open- >> mpi.org] >>>> On Behalf Of Sébastien Boisvert >>>> Sent: Friday, September 28, 2012 1:22 AM >>>> To: us...@open-mpi.org >>>> Subject: [OMPI users] About MPI_TAG_UB >>>> >>>> Hello, >>>> >>>> I am running Ray (a distributed software in genomics) with Open-MPI >>>> on >>>> 2048 processes and everything runs fine. Ray has a any-to-any >>>> communication pattern. >>>> To avoid using too much memory, I implemented a virtual message >> router. >>>> >>>> Without the virtual message router, I get messages like these: >>>> >>>> >> [cp2558][[30209,1],0][connect/btl_openib_connect_oob.c:490:qp_create_ >>>> o ne] error creating qp errno says Cannot allocate memory >>>> >>>> We did some tests on the Cray XE6 on 4096 processing elements (4096 >>>> MPI >>>> ranks) without the virtual message router and everything runs fine as is. >>> So >>>> using the virtual message router is not required. >>>> >>>> The real message tag, the real source and the real destination are >>>> stored >>> in >>>> the MPI tag. I know, this is ugly, but it works. I can not store this >>> information >>>> in the message buffer because the buffer can be NULL. >>>> >>>> bits 0 to 7: tag (8 bits, values from 0 to 255, 256 possible values) >>>> bits >>> 8 to 19: >>>> true source (12 bits, values from 0 to 4095, 4096 possible values) >>>> bits 20 >>> to 31: >>>> true destination (12 bits, values from 0 to 4095, 4096 possible >>>> values) >>>> >>>> Without the virtual router, my code is compliant with the fact that >>>> MPI_Comm_get_attr(MPI_COMM_WORLD, MPI_TAG_UB,...) is at least >>>> 32767 (my tags are <= 255). >>>> >>>> >>>> When I try jobs with 4096 processes with the virtual message router, >>>> I get >>> the >>>> error: >>>> >>>> MPI_ERR_TAG: invalid tag. >>>> >>>> Without the virtual message router I get: >>>> >>>> >> [cp2558][[30209,1],0][connect/btl_openib_connect_oob.c:490:qp_create_ >>>> o ne] error creating qp errno says Cannot allocate memory >>>> >>>> With Open-MPI 1.5.4, the upper bound is 17438272 (at least in our build). >>>> That explains MPI_ERR_TAG. >>>> >>>> >>>> My 2 questions: >>>> >>>> 1. Is there a better way to store routing information ? >>>> >>>> 2. Can I create my own communicator and set its MPI_TAG_UB to >>>> whatever I want ? >>>> >>>> >>>> Thanks ! >>>> >>>> *** >>>> Sébastien Boisvert >>>> Ph.D. student >>>> http://boisvert.info/ >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users
