FYI:
I opened:
https://github.com/open-mpi/ompi/issues/638
to track this.
Steve.
On 6/10/2015 4:07 PM, Ralph Castain wrote:
Done
On Jun 10, 2015, at 1:55 PM, Steve Wise <sw...@opengridcomputing.com
<mailto:sw...@opengridcomputing.com>> wrote:
If you're trying to release 1.8.6, I recommend you revert the change
that breaks things until we can figure this out.
-----Original Message-----
From: Jeff Squyres (jsquyres) [mailto:jsquy...@cisco.com]
Sent: Wednesday, June 10, 2015 3:30 PM
To: Open MPI User's List
Cc: Nathan Hjelm; Steve Wise
Subject: Re: [OMPI users] Default value of btl_openib_memalign_threshold
Nathan / Steve -- you guys are nominally the owners of the openib
BTL: can you please investigate?
On Jun 10, 2015, at 4:15 PM, Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>> wrote:
Odd - without that setting, the value is essentially undefined, so
it’s hard to understand how that is any better. Maybe the whole
alignment thing is busted, and leaving it undefined (which usually
defaults to zero, but not always) causes it to be turned “off”?
I don’t really care, mind you - but it is clearly an error the way
it was before.
On Jun 10, 2015, at 12:39 PM, Jeff Squyres (jsquyres)
<jsquy...@cisco.com <mailto:jsquy...@cisco.com>> wrote:
Ralph --
This change was not correct
(https://github.com/open-mpi/ompi/commit/ce915b5757d428d3e914dcef50bd4b2636561bca).
It is
causing memory corruption in the openib BTL.
On May 25, 2015, at 11:56 AM, Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>> wrote:
I don’t see a problem with it. FWIW: I’m getting ready to release
1.8.6 in the next week
On May 25, 2015, at 8:46 AM, Xavier Besseron
<xavier.besse...@uni.lu <mailto:xavier.besse...@uni.lu>> wrote:
Good that it will be fixed in the next release!
In the meantime, and because it might impact other users,
I would like to ask my sysadmins to set
btl_openib_memalign_threshold=12288 in
etc/openmpi-mca-params.conf on our clusters.
Do you see any good reason not doing it?
Thanks!
Xavier
On Mon, May 25, 2015 at 4:12 PM, Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>> wrote:
I found the problem - someone had a typo in btl_openib_mca.c.
The threshold need to be set to the module eager limit as that
is the
only thing defined at that point.
Thanks for bringing it to our attention! I’ll set it up to go
into 1.8.6
On May 25, 2015, at 3:04 AM, Xavier Besseron
<xavier.besse...@uni.lu <mailto:xavier.besse...@uni.lu>> wrote:
Hi,
Thanks for your reply Ralph.
The option only I'm using when configuring OpenMPI is '--prefix'.
When checking the config.log file, I see
configure:208504: checking whether the openib BTL will use
malloc hooks
configure:208510: result: yes
so I guess it is properly enabled (full config.log in
attachment of this email).
However, I think I have the reason of the bug (lines refer to
source code of OpenMPI 1.8.5):
The default value of memalign_threshold is taken from
eager_limit in function btl_openib_register_mca_params() in
btl_openib_mca.c line 717.
But the default value is eager_limit is set in
btl_openib_component.c at line 193 right after the call to
btl_openib_register_mca_params().
To summarize, memalign_threshold gets its value from
eager_limit before this one gets its value assigned.
Best regards,
Xavier
On Mon, May 25, 2015 at 2:27 AM, Ralph Castain
<r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
Looking at the code, we do in fact set the memalign_threshold =
eager_limit by default, but only if you configured with —enable-
btl-openib-malloc-alignment AND/OR we found the malloc hook
functions were available.
You might check config.log to see if the openib malloc hooks
were enabled. My guess is that they weren’t, for some reason.
On May 24, 2015, at 9:07 AM, Xavier Besseron
<xavier.besse...@uni.lu <mailto:xavier.besse...@uni.lu>> wrote:
Dear OpenMPI developers / users,
This is much more a comment than a question since I believe I
have already solved my issue. But I would like to report it.
I have noticed my code performed very badly with OpenMPI when
Infinand is enabled, sometime +50% or even +100% overhead.
I also have this slowdown when running with one thread and one
process. In such case, there is no other MPI call than MPI_Init()
and MPI_Finalize().
This overhead disappears if I disable at runtime the openib
btl, ie with '--mca btl ^openib'.
After further investigation, I figured out it comes from the
memory allocator which is aligning every memory allocation when
Infiniband is used.
This makes sense because my code is a large irregular C++ code
creating and deleting many objects.
Just below is the documentation of the relevant MCA parameters
coming ompi_info:
MCA btl: parameter "btl_openib_memalign" (current value: "32",
data source: default, level: 9 dev/all, type: int)
[64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0)
memoryalignment for all malloc calls if btl openib is used.
MCA btl: parameter "btl_openib_memalign_threshold" (current
value: "0", data source: default, level: 9 dev/all, type: size_t)
Allocating memory more than
btl_openib_memalign_threshholdbytes will automatically be
algined to the value of
btl_openib_memalign bytes.memalign_threshhold defaults to the same
value as mca_btl_openib_eager_limit.
MCA btl: parameter "btl_openib_eager_limit" (current value:
"12288", data source: default, level: 4 tuner/basic, type: size_t)
Maximum size (in bytes, including header) of "short"
messages (must be >= 1).
In the end, the problem is that the default value for
btl_openib_memalign_threshold is 0, which means that all
memory allocations
are aligned to 32 bits.
The documentation says that the default value of
btl_openib_memalign_threshold should be the the same as
btl_openib_eager_limit, ie 12288 instead of 0.
On my side, changing btl_openib_memalign_threshold to 12288
fixes my performance issue.
However, I believe that the default value of
btl_openib_memalign_threshold should be fixed in the OpenMPI
code (or at least the
documentation should be fixed).
I tried OpenMPI 1.8.5, 1.7.3 and 1.6.4 and it's all the same.
Bonus question:
As this issue might impact other users, I'm considering
applying a global fix on our clusters by setting this default
value
etc/openmpi-mca-params.conf.
Do you see any good reason not doing it?
Thank you for your comments.
Best regards,
Xavier
--
Dr Xavier BESSERON
Research associate
FSTC, University of Luxembourg
Campus Kirchberg, Office E-007
Phone: +352 46 66 44 5418
http://luxdem.uni.lu/
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/05/26913.php
--
Dr Xavier BESSERON
Research associate
FSTC, University of Luxembourg
Campus Kirchberg, Office E-007
Phone: +352 46 66 44 5418
http://luxdem.uni.lu/
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/05/26915.php
--
Dr Xavier BESSERON
Research associate
FSTC, University of Luxembourg
Campus Kirchberg, Office E-007
Phone: +352 46 66 44 5418
http://luxdem.uni.lu/
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/05/26918.php
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/05/26920.php
--
Jeff Squyres
jsquy...@cisco.com <mailto:jsquy...@cisco.com>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/06/27086.php
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/06/27087.php
--
Jeff Squyres
jsquy...@cisco.com <mailto:jsquy...@cisco.com>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
users mailing list
us...@open-mpi.org <mailto:us...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/06/27089.php
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/06/27090.php
