Public bug reported:
This is a follow-up issue for LP#1896125.
This problem has happened when LDAP connection pooling is on
(use_pool=True), page_size > 0 and pool_connection_timeout is < 'ldap
server response time'. The scenario is as follows:
- An user tries to log in to a domain that is attached to LDAP backend.
- LDAP server does not respond in `pool_connection_timeout` seconds, causing
LDAP connection to raise a ldap.TIMEOUT() exception
- From now on, all subsequent LDAP requests will fail with
ldappool.MaxConnectionReachedError
An in-depth analysis explains why it happens:
- LDAP query initiated for user login request with BaseLdap._ldap_get()
function call, which grabs a connection with self.get_connection() and invokes
conn.search_s()
- conn.search_s() invokes conn._paged_search_s() since page_size is > 0
- conn._paged_search_s() calls conn.search_ext() (PooledLDAPHandler.search_ext)
method
- conn.search_ext() initiates an asynchronous LDAP request and returns an
AsynchronousMessage object to the _paged_search_s(), representing the request.
- conn._paged_search_s() tries to obtain asynchronous LDAP request results via
calling conn.result3() (PooledLDAPHandler.result3)
- conn.result3() calls message.connection.result3()
- the server cannot respond in pool_connection_timeout seconds,
- message.connection.result3() raises a ldap.TIMEOUT(), causes subsequent
connection release function, message.clean() to be not called
- the connection is kept active forever, subsequent requests cannot use it
anymore
Reproducer:
- Deploy an LDAP server of your choice
- Fill it with many data so the search takes more than
`pool_connection_timeout` seconds
- Define a keystone domain with the LDAP driver with following options:
[ldap]
use_pool = True
page_size = 100
pool_connection_timeout = 3
pool_retry_max = 3
pool_size = 10
- Point the domain to the LDAP server
- Try to login to the OpenStack dashboard, or try to do anything that uses the
LDAP user
- Observe the /var/log/apache2/keystone_error.log, it should contain
ldap.TIMEOUT() stack traces followed by `ldappool.MaxConnectionReachedError`
stack traces
Known workarounds:
- Disable LDAP pooling by setting use_pool=Flase
- Set page_size to 0
** Affects: keystone
Importance: Undecided
Assignee: Mustafa Kemal Gilor (mustafakemalgilor)
Status: New
** Tags: sts
** Summary changed:
- PooledLDAPHandler.result3 does not release pool connection back when an
exception raises
+ PooledLDAPHandler.result3 does not release pool connection back when an
exception is raised
** Changed in: keystone
Assignee: (unassigned) => Mustafa Kemal Gilor (mustafakemalgilor)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1998789
Title:
PooledLDAPHandler.result3 does not release pool connection back when
an exception is raised
Status in OpenStack Identity (keystone):
New
Bug description:
This is a follow-up issue for LP#1896125.
This problem has happened when LDAP connection pooling is on
(use_pool=True), page_size > 0 and pool_connection_timeout is < 'ldap
server response time'. The scenario is as follows:
- An user tries to log in to a domain that is attached to LDAP backend.
- LDAP server does not respond in `pool_connection_timeout` seconds, causing
LDAP connection to raise a ldap.TIMEOUT() exception
- From now on, all subsequent LDAP requests will fail with
ldappool.MaxConnectionReachedError
An in-depth analysis explains why it happens:
- LDAP query initiated for user login request with BaseLdap._ldap_get()
function call, which grabs a connection with self.get_connection() and invokes
conn.search_s()
- conn.search_s() invokes conn._paged_search_s() since page_size is > 0
- conn._paged_search_s() calls conn.search_ext()
(PooledLDAPHandler.search_ext) method
- conn.search_ext() initiates an asynchronous LDAP request and returns an
AsynchronousMessage object to the _paged_search_s(), representing the request.
- conn._paged_search_s() tries to obtain asynchronous LDAP request results
via calling conn.result3() (PooledLDAPHandler.result3)
- conn.result3() calls message.connection.result3()
- the server cannot respond in pool_connection_timeout seconds,
- message.connection.result3() raises a ldap.TIMEOUT(), causes subsequent
connection release function, message.clean() to be not called
- the connection is kept active forever, subsequent requests cannot use it
anymore
Reproducer:
- Deploy an LDAP server of your choice
- Fill it with many data so the search takes more than
`pool_connection_timeout` seconds
- Define a keystone domain with the LDAP driver with following options:
[ldap]
use_pool = True
page_size = 100
pool_connection_timeout = 3
pool_retry_max = 3
pool_size = 10
- Point the domain to the LDAP server
- Try to login to the OpenStack dashboard, or try to do anything that uses
the LDAP user
- Observe the /var/log/apache2/keystone_error.log, it should contain
ldap.TIMEOUT() stack traces followed by `ldappool.MaxConnectionReachedError`
stack traces
Known workarounds:
- Disable LDAP pooling by setting use_pool=Flase
- Set page_size to 0
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1998789/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
