Public bug reported:
With wsgi_default_pool_size=100[1], if the keepalived state change
server gets 100 requests at the same time, while processing the requests
heavy load is seen on cpu, making the network node unresponsive. For
each request, keepalived state change server spawns a new meta data
proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load,
with "top" command, I can see many metadata proxy processes in "running"
state at same time(see the attachment).
When wsgi_default_pool_size=8, I see state change server spawning 8
metadata proxy processes at a time("top" command shows 8 meta data proxy
processes in "running" state at a time), cpu load is less and metadata
proxy processes(for example, 100) spawned for all requests without
failures.
We can keep wsgi_default_pool_size=100 for neutron API server, and use
seperate configuration for UnixDomainWSGIServer(for example
CONF.unix_domain_wsgi_default_pool_size).
neutron/agent/linux/utils.py
class UnixDomainWSGIServer(wsgi.Server):
def _run(self, application, socket):
"""Start a WSGI service in a new green thread."""
logger = logging.getLogger('eventlet.wsgi.server')
eventlet.wsgi.server(socket,
application,
max_size=CONF.unix_domain_wsgi_default_pool_size,
protocol=UnixDomainHttpProtocol,
log=logger)
[1]
https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807
** Affects: neutron
Importance: Undecided
Assignee: venkata anil (anil-venkata)
Status: New
** Tags: l3-ha
** Attachment added: "top.gif"
https://bugs.launchpad.net/bugs/1581580/+attachment/4662237/+files/top.gif
** Changed in: neutron
Assignee: (unassigned) => venkata anil (anil-venkata)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1581580
Title:
Heavy cpu load seen when keepalived state change server gets
wsi_default_pool_size requests at same time
Status in neutron:
New
Bug description:
With wsgi_default_pool_size=100[1], if the keepalived state change
server gets 100 requests at the same time, while processing the
requests heavy load is seen on cpu, making the network node
unresponsive. For each request, keepalived state change server spawns
a new meta data proxy process(i.e neutron-ns-metadata-proxy). During
heavy cpu load, with "top" command, I can see many metadata proxy
processes in "running" state at same time(see the attachment).
When wsgi_default_pool_size=8, I see state change server spawning 8
metadata proxy processes at a time("top" command shows 8 meta data
proxy processes in "running" state at a time), cpu load is less and
metadata proxy processes(for example, 100) spawned for all requests
without failures.
We can keep wsgi_default_pool_size=100 for neutron API server, and use
seperate configuration for UnixDomainWSGIServer(for example
CONF.unix_domain_wsgi_default_pool_size).
neutron/agent/linux/utils.py
class UnixDomainWSGIServer(wsgi.Server):
def _run(self, application, socket):
"""Start a WSGI service in a new green thread."""
logger = logging.getLogger('eventlet.wsgi.server')
eventlet.wsgi.server(socket,
application,
max_size=CONF.unix_domain_wsgi_default_pool_size,
protocol=UnixDomainHttpProtocol,
log=logger)
[1]
https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1581580/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
