On 03/26/2015 04:07 PM, Ian Wienand wrote:
> See [1] for some more details; but the short story is that the various
> swift processes -- even just sitting around freshly installed from
> devstack before anything happens -- take up twice as much space on
> centos as ubuntu
>
> --- swift (% total system memory) ---
> ubuntu 6.6%
> centos 12%
So after more investigation, it turns out that pyOpenSSL has rewritten
itself in python; necessitating dependencies on the "cryptography"
package and cffi & pycparser [1]. Examining the heap shows where the
memory has gone missing :
Partition of a set of 205366 objects. Total size = 30969040 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 67041 33 5712560 18 5712560 18 str
1 10260 5 2872800 9 8585360 28 dict of pycparser.plyparser.Coord
2 27765 14 2367552 8 10952912 35 tuple
3 1215 1 2246760 7 13199672 43 dict (no owner)
4 1882 1 1972336 6 15172008 49 dict of pycparser.c_ast.Decl
5 16085 8 1736232 6 16908240 55 list
6 360 0 1135296 4 18043536 58 dict of module
7 4041 2 1131480 4 19175016 62 dict of pycparser.c_ast.TypeDecl
8 4021 2 1125880 4 20300896 66 dict of
pycparser.c_ast.IdentifierType
9 6984 3 893952 3 21194848 68 types.CodeType
<413 more rows. Type e.g. '_.more' to view.>
If I reinstall the packaged version of pyOpenSSL, all that drops out
and we're back to a more reasonable usage
Partition of a set of 95591 objects. Total size = 12500080 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 45837 48 3971040 32 3971040 32 str
1 22843 24 1943416 16 5914456 47 tuple
2 298 0 978160 8 6892616 55 dict of module
3 6065 6 776320 6 7668936 61 types.CodeType
4 551 1 742184 6 8411120 67 dict (no owner)
5 805 1 725520 6 9136640 73 type
6 5876 6 705120 6 9841760 79 function
7 805 1 666232 5 10507992 84 dict of type
8 289 0 279832 2 10787824 86 dict of class
9 152 0 159296 1 10947120 88 dict of pkg_resources.Distribution
<310 more rows. Type e.g. '_.more' to view.>
The end result of this is that swift-* processes go from consuming
about 6% of a CI VM's 8gb to 12%. This 500mb is enough to push the
host into OOM when tempest gets busy. For more see [2]. a workaround is [3]
I'll spend a bit more time on this -- I haven't determined if it's
centos or swift specific yet -- but in the mean-time, beware of
recent pyOpenSSL
-i
[1]
https://github.com/pyca/pyopenssl/commit/fd193a2f9dd8be80d9f42d8dd8068de5f5ac5e67
[2] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[3] https://review.openstack.org/#/c/168217/
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
