Done
> On Jun 10, 2015, at 1:55 PM, Steve Wise <sw...@opengridcomputing.com> wrote: > > If you're trying to release 1.8.6, I recommend you revert the change that > breaks things until we can figure this out. > >> -----Original Message----- >> From: Jeff Squyres (jsquyres) [mailto:jsquy...@cisco.com] >> Sent: Wednesday, June 10, 2015 3:30 PM >> To: Open MPI User's List >> Cc: Nathan Hjelm; Steve Wise >> Subject: Re: [OMPI users] Default value of btl_openib_memalign_threshold >> >> Nathan / Steve -- you guys are nominally the owners of the openib BTL: can >> you please investigate? >> >> >>> On Jun 10, 2015, at 4:15 PM, Ralph Castain <r...@open-mpi.org> wrote: >>> >>> Odd - without that setting, the value is essentially undefined, so it’s >>> hard to understand how that is any better. Maybe the whole >> alignment thing is busted, and leaving it undefined (which usually defaults >> to zero, but not always) causes it to be turned “off”? >>> >>> I don’t really care, mind you - but it is clearly an error the way it was >>> before. >>> >>> >>>> On Jun 10, 2015, at 12:39 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> >>>> wrote: >>>> >>>> Ralph -- >>>> >>>> This change was not correct >>>> (https://github.com/open-mpi/ompi/commit/ce915b5757d428d3e914dcef50bd4b2636561bca). >>>> It is >> causing memory corruption in the openib BTL. >>>> >>>> >>>> >>>>> On May 25, 2015, at 11:56 AM, Ralph Castain <r...@open-mpi.org> wrote: >>>>> >>>>> I don’t see a problem with it. FWIW: I’m getting ready to release 1.8.6 >>>>> in the next week >>>>> >>>>> >>>>>> On May 25, 2015, at 8:46 AM, Xavier Besseron <xavier.besse...@uni.lu> >>>>>> wrote: >>>>>> >>>>>> Good that it will be fixed in the next release! >>>>>> >>>>>> In the meantime, and because it might impact other users, >>>>>> I would like to ask my sysadmins to set >>>>>> btl_openib_memalign_threshold=12288 in etc/openmpi-mca-params.conf on >>>>>> our clusters. >>>>>> >>>>>> Do you see any good reason not doing it? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> Xavier >>>>>> >>>>>> >>>>>> >>>>>> On Mon, May 25, 2015 at 4:12 PM, Ralph Castain <r...@open-mpi.org> wrote: >>>>>> I found the problem - someone had a typo in btl_openib_mca.c. The >>>>>> threshold need to be set to the module eager limit as that is the >> only thing defined at that point. >>>>>> >>>>>> Thanks for bringing it to our attention! I’ll set it up to go into 1.8.6 >>>>>> >>>>>> >>>>>>> On May 25, 2015, at 3:04 AM, Xavier Besseron <xavier.besse...@uni.lu> >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Thanks for your reply Ralph. >>>>>>> >>>>>>> The option only I'm using when configuring OpenMPI is '--prefix'. >>>>>>> When checking the config.log file, I see >>>>>>> >>>>>>> configure:208504: checking whether the openib BTL will use malloc hooks >>>>>>> configure:208510: result: yes >>>>>>> >>>>>>> so I guess it is properly enabled (full config.log in attachment of >>>>>>> this email). >>>>>>> >>>>>>> >>>>>>> >>>>>>> However, I think I have the reason of the bug (lines refer to source >>>>>>> code of OpenMPI 1.8.5): >>>>>>> >>>>>>> The default value of memalign_threshold is taken from eager_limit in >>>>>>> function btl_openib_register_mca_params() in >> btl_openib_mca.c line 717. >>>>>>> But the default value is eager_limit is set in btl_openib_component.c >>>>>>> at line 193 right after the call to >> btl_openib_register_mca_params(). >>>>>>> >>>>>>> To summarize, memalign_threshold gets its value from eager_limit before >>>>>>> this one gets its value assigned. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Xavier >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, May 25, 2015 at 2:27 AM, Ralph Castain <r...@open-mpi.org> >>>>>>> wrote: >>>>>>> Looking at the code, we do in fact set the memalign_threshold = >>>>>>> eager_limit by default, but only if you configured with —enable- >> btl-openib-malloc-alignment AND/OR we found the malloc hook functions were >> available. >>>>>>> >>>>>>> You might check config.log to see if the openib malloc hooks were >>>>>>> enabled. My guess is that they weren’t, for some reason. >>>>>>> >>>>>>> >>>>>>>> On May 24, 2015, at 9:07 AM, Xavier Besseron <xavier.besse...@uni.lu> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Dear OpenMPI developers / users, >>>>>>>> >>>>>>>> This is much more a comment than a question since I believe I have >>>>>>>> already solved my issue. But I would like to report it. >>>>>>>> >>>>>>>> I have noticed my code performed very badly with OpenMPI when Infinand >>>>>>>> is enabled, sometime +50% or even +100% overhead. >>>>>>>> I also have this slowdown when running with one thread and one >>>>>>>> process. In such case, there is no other MPI call than MPI_Init() >> and MPI_Finalize(). >>>>>>>> This overhead disappears if I disable at runtime the openib btl, ie >>>>>>>> with '--mca btl ^openib'. >>>>>>>> After further investigation, I figured out it comes from the memory >>>>>>>> allocator which is aligning every memory allocation when >> Infiniband is used. >>>>>>>> This makes sense because my code is a large irregular C++ code >>>>>>>> creating and deleting many objects. >>>>>>>> >>>>>>>> Just below is the documentation of the relevant MCA parameters coming >>>>>>>> ompi_info: >>>>>>>> >>>>>>>> MCA btl: parameter "btl_openib_memalign" (current value: "32", data >>>>>>>> source: default, level: 9 dev/all, type: int) >>>>>>>> [64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0) >>>>>>>> memoryalignment for all malloc calls if btl openib is used. >>>>>>>> >>>>>>>> MCA btl: parameter "btl_openib_memalign_threshold" (current value: >>>>>>>> "0", data source: default, level: 9 dev/all, type: size_t) >>>>>>>> Allocating memory more than btl_openib_memalign_threshholdbytes >>>>>>>> will automatically be algined to the value of >> btl_openib_memalign bytes.memalign_threshhold defaults to the same value as >> mca_btl_openib_eager_limit. >>>>>>>> >>>>>>>> MCA btl: parameter "btl_openib_eager_limit" (current value: "12288", >>>>>>>> data source: default, level: 4 tuner/basic, type: size_t) >>>>>>>> Maximum size (in bytes, including header) of "short" messages >>>>>>>> (must be >= 1). >>>>>>>> >>>>>>>> >>>>>>>> In the end, the problem is that the default value for >>>>>>>> btl_openib_memalign_threshold is 0, which means that all memory >>>>>>>> allocations >> are aligned to 32 bits. >>>>>>>> The documentation says that the default value of >>>>>>>> btl_openib_memalign_threshold should be the the same as >> btl_openib_eager_limit, ie 12288 instead of 0. >>>>>>>> >>>>>>>> On my side, changing btl_openib_memalign_threshold to 12288 fixes my >>>>>>>> performance issue. >>>>>>>> However, I believe that the default value of >>>>>>>> btl_openib_memalign_threshold should be fixed in the OpenMPI code (or >>>>>>>> at least the >> documentation should be fixed). >>>>>>>> >>>>>>>> I tried OpenMPI 1.8.5, 1.7.3 and 1.6.4 and it's all the same. >>>>>>>> >>>>>>>> >>>>>>>> Bonus question: >>>>>>>> As this issue might impact other users, I'm considering applying a >>>>>>>> global fix on our clusters by setting this default value >> etc/openmpi-mca-params.conf. >>>>>>>> Do you see any good reason not doing it? >>>>>>>> >>>>>>>> Thank you for your comments. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> Xavier >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dr Xavier BESSERON >>>>>>>> Research associate >>>>>>>> FSTC, University of Luxembourg >>>>>>>> Campus Kirchberg, Office E-007 >>>>>>>> Phone: +352 46 66 44 5418 >>>>>>>> http://luxdem.uni.lu/ >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/users/2015/05/26913.php >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dr Xavier BESSERON >>>>>>> Research associate >>>>>>> FSTC, University of Luxembourg >>>>>>> Campus Kirchberg, Office E-007 >>>>>>> Phone: +352 46 66 44 5418 >>>>>>> http://luxdem.uni.lu/ >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/05/26915.php >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dr Xavier BESSERON >>>>>> Research associate >>>>>> FSTC, University of Luxembourg >>>>>> Campus Kirchberg, Office E-007 >>>>>> Phone: +352 46 66 44 5418 >>>>>> http://luxdem.uni.lu/ >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2015/05/26918.php >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2015/05/26920.php >>>> >>>> >>>> -- >>>> Jeff Squyres >>>> jsquy...@cisco.com >>>> For corporate legal information go to: >>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2015/06/27086.php >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2015/06/27087.php >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27089.php > <http://www.open-mpi.org/community/lists/users/2015/06/27089.php>
