Following up on my own question, in case it's useful to others....
Turns out that keystonemiddleware uses eventlet, and, by default,
creates a connection to memcached from each green thread (and doesn't
clean them up), and the green threads are essentially unlimited.
There is a solution for this, which implements a shared connection pool.
It's enabled via the keystone_authtoken.memcache_use_advanced_pool
config option.
Unfortunately it was broken in a few different ways (I guess this means
that no one is using it?)
I've worked with the keystone devs, and we were able to get a fix (in
keystonemiddleware) in just in time for the Rocky release. Related fixes
have also been backported to Queens (for the next update), and a couple
needed for Pike are pending completion.
With this in place, so-far I have not seen more than one connection to
memcached for each neutron-api worker process, and everything seems to
be working well.
Some relevant changes:
master:
https://review.openstack.org/#/c/583695/
Queens:
https://review.openstack.org/#/c/583698/
https://review.openstack.org/#/c/583684/
Pike:
https://review.openstack.org/#/c/583699/
https://review.openstack.org/#/c/583835/
I do wonder how others are managing memcached connections for larger
deployments...
~iain
On 06/26/2018 12:59 PM, iain MacDonnell wrote:
In diagnosing a situation where a Pike deployment was intermittently
slower (in general), I discovered that it was (sometimes) exceeding
memcached's maximum connection limit, which is set to 4096.
Looking closer, ~2750 of the connections are from 8 neutron-server
process. neutron-server is configured with 8 API workers, and those 8
processes have a combined total of ~2750 connections to memcached:
# lsof -i TCP:11211 | awk '/^neutron-s/ {print $2}' | sort | uniq -c
245 2611
306 2612
228 2613
406 2614
407 2615
385 2616
369 2617
398 2618
#
There doesn't seem to be much turnover - comparing samples of the
connections (incl. source port) 15 mins apart, two were dropped, and one
new one added.
In neutron.conf, keystone_authtoken.memcached_servers is configured, but
nothing else pertaining to caching, so
keystone_authtoken.memcache_pool_maxsize should default to 10.
Am I misunderstanding something, or shouldn't I see a maximum of 10
connections from each of the neutron-server API workers, with this
configuration?
Any known issues, or pointers to what I'm missing?
TIA,
~iain
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
