Hi, We've been having an issue with Centos 7 jobs and the host running out of memory. After some investigation; one likely angle seems to be the memory usage by swift.
See [1] for some more details; but the short story is that the various swift processes -- even just sitting around freshly installed from devstack before anything happens -- take up twice as much space on centos as ubuntu --- swift (% total system memory) --- ubuntu 6.6% centos 12% In general memory usage is higher on centos, but this one is an outlier. Unfortunately, the main difference between the two appears to be heap allocations (see comments in [1]) which doesn't give a lot of clues. The end result is that the host ends up just plain running out of memory; the OOM killer kicks in and then everything starts collapsing. I had the host sending me telemetry while it was running; the last entry before things fell over was [2] and we can see that it's not just one thing that comes along and sucks up memory, but death by a thousand cuts. I think the Centos 7 job is struggling to fit into the 8gb available so we're susceptible to finding memory issues first. Any swift people have some ideas on this? Thanks -i [1] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job [2] http://paste.openstack.org/show/196769/ __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
