Public bug reported: [Impact]
If you connect to a slapd server using TLS and for some reason, the connection hangs or fails to completely establish, TIMEOUT and NETWORK_TIMEOUT values do not function as they are intended. They get ignored, and the connecting process e.g. ldapsearch just sits there at 100% CPU waiting forever. If slapd is set up in High Availability with servers to failover to, the failover will never occur, due to the connecting processes sitting there at 100% CPU waiting for the connection that will never succeed, leading to outages. You can simulate this easily with: $ ldapsearch -x -H ldaps://example.com $ sudo kill -STOP <slapd pid> $ ldapsearch -x -H ldaps://example.com and the last call will hang forever. There doesn't seem to be any workaround for this, apart from not using SSL. [Testcase] You can get away with just the one VM for this. Set up the ldap server (from server docs): https://ubuntu.com/server/docs/install-and-configure-ldap $ sudo apt install slapd ldap-utils $ sudo dpkg-reconfigure slapd Set DNS to example.com $ ldapsearch -x -LLL -H ldap:/// -b dc=example,dc=com dn dn: dc=example,dc=com $ vim add_content.ldif dn: ou=People,dc=example,dc=com objectClass: organizationalUnit ou: People dn: ou=Groups,dc=example,dc=com objectClass: organizationalUnit ou: Groups dn: cn=miners,ou=Groups,dc=example,dc=com objectClass: posixGroup cn: miners gidNumber: 5000 dn: uid=john,ou=People,dc=example,dc=com objectClass: inetOrgPerson objectClass: posixAccount objectClass: shadowAccount uid: john sn: Doe givenName: John cn: John Doe displayName: John Doe uidNumber: 10000 gidNumber: 5000 userPassword: {CRYPT}x gecos: John Doe loginShell: /bin/bash homeDirectory: /home/john $ ldapadd -x -D cn=admin,dc=example,dc=com -W -f add_content.ldif $ ldapsearch -x -LLL -b dc=example,dc=com '(uid=john)' cn gidNumber dn: uid=john,ou=People,dc=example,dc=com cn: John Doe gidNumber: 5000 $ ldappasswd -x -D cn=admin,dc=example,dc=com -W -S uid=john,ou=people,dc=example,dc=com New password: Re-enter new password: Enter LDAP Password: Configure the server for SSL (from server docs): https://ubuntu.com/server/docs/ldap-and-transport-layer-security-tls $ sudo apt install gnutls-bin ssl-cert $ sudo certtool --generate-privkey --bits 4096 --outfile /etc/ssl/private/mycakey.pem $ sudo vim /etc/ssl/ca.info cn = Example Company ca cert_signing_key expiration_days = 3650 $ sudo certtool --generate-self-signed \ --load-privkey /etc/ssl/private/mycakey.pem \ --template /etc/ssl/ca.info \ --outfile /usr/local/share/ca-certificates/mycacert.crt $ sudo update-ca-certificates $ sudo certtool --generate-privkey \ --bits 2048 \ --outfile /etc/ldap/ldap01_slapd_key.pem $ sudo vim /etc/ssl/ldap01.info organization = Example Company cn = example.com tls_www_server encryption_key signing_key expiration_days = 365 $ sudo certtool --generate-certificate \ --load-privkey /etc/ldap/ldap01_slapd_key.pem \ --load-ca-certificate /etc/ssl/certs/mycacert.pem \ --load-ca-privkey /etc/ssl/private/mycakey.pem \ --template /etc/ssl/ldap01.info \ --outfile /etc/ldap/ldap01_slapd_cert.pem $ sudo chgrp openldap /etc/ldap/ldap01_slapd_key.pem $ sudo chmod 0640 /etc/ldap/ldap01_slapd_key.pem $ sudo vim certinfo.ldif dn: cn=config add: olcTLSCACertificateFile olcTLSCACertificateFile: /etc/ssl/certs/mycacert.pem - add: olcTLSCertificateFile olcTLSCertificateFile: /etc/ldap/ldap01_slapd_cert.pem - add: olcTLSCertificateKeyFile olcTLSCertificateKeyFile: /etc/ldap/ldap01_slapd_key.pem $ sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f certinfo.ldif $ sudo vim /etc/default/slapd Change LAPD_SERVICES="ldap:/// ldapi:///" to SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///" Next add TIMEOUT and NETWORK_TIMEOUT to the config file. $ sudo vim /etc/ldap/ldap.conf NETWORK_TIMEOUT 30 TIMEOUT 30 $ sudo systemctl restart slapd Reproduce the problem: $ ldapsearch -x -H ldaps://example.com # extended LDIF # # LDAPv3 # base <> (default) with scope subtree # filter: (objectclass=*) # requesting: ALL # # search result search: 2 result: 32 No such object # numResponses: 1 $ ps aux | grep slapd openldap 4097 0.0 0.1 1224136 7876 ? Ssl 02:22 0:00 /usr/sbin/slapd -h ldap:/// ldapi:/// ldaps:/// -g openldap -u openldap -F /etc/ldap/slapd.d $ sudo kill -STOP 4097 $ ldapsearch -x -H ldaps://example.com Check top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4237 ubuntu 20 0 20724 10956 6656 R 100.0 0.3 0:35.70 ldapsearch wait for 30 seconds, see nothing changes. If you install packages from the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf401397-test You should see the timeout work as expected, and the ldapsearch process will give up after 30 seconds. [Where problems could occur] We are changing how connections are tracked within openldap to better accurately say what state we are in, and for the timeouts to work when the connection doesn't make any progress. This will cause TIMEOUT and NETWORK_TIMEOUT to start working as intended, which might surprise some sysadmins who configured it, and likely forgot about it when they never had any timeout issues. If a regression were to occur, it would affect initial connections to openldap servers, or even long running commands that might exceed the TIMEOUT on queries over large ldap databases. [Other info] Upstream issues: https://bugs.openldap.org/show_bug.cgi?id=8047 https://bugs.openldap.org/show_bug.cgi?id=10141 This was fixed in 2.6.9 upstream by: commit d143f7a2dc82fb66e7741b93a1ae9e874ce2ac46 Author: Ondřej Kuzník <on...@mistotebe.net> Date: Mon, 21 Oct 2024 11:50:11 +0100 Subject: ITS#8047 Fix TLS connection timeout handling Link: https://git.openldap.org/openldap/openldap/-/commit/d143f7a2dc82fb66e7741b93a1ae9e874ce2ac46 This fix is required in all releases. ** Affects: openldap (Ubuntu) Importance: Medium Status: In Progress ** Affects: openldap (Ubuntu Focal) Importance: Medium Assignee: Matthew Ruffell (mruffell) Status: In Progress ** Affects: openldap (Ubuntu Jammy) Importance: Medium Assignee: Matthew Ruffell (mruffell) Status: In Progress ** Affects: openldap (Ubuntu Noble) Importance: Medium Assignee: Matthew Ruffell (mruffell) Status: In Progress ** Affects: openldap (Ubuntu Oracular) Importance: Medium Assignee: Matthew Ruffell (mruffell) Status: In Progress ** Affects: openldap (Ubuntu Plucky) Importance: Medium Status: In Progress ** Tags: sts ** Also affects: openldap (Ubuntu Plucky) Importance: Undecided Status: New ** Also affects: openldap (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: openldap (Ubuntu Oracular) Importance: Undecided Status: New ** Also affects: openldap (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: openldap (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: openldap (Ubuntu Focal) Status: New => In Progress ** Changed in: openldap (Ubuntu Jammy) Status: New => In Progress ** Changed in: openldap (Ubuntu Noble) Status: New => In Progress ** Changed in: openldap (Ubuntu Oracular) Status: New => In Progress ** Changed in: openldap (Ubuntu Plucky) Status: New => In Progress ** Changed in: openldap (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: openldap (Ubuntu Jammy) Importance: Undecided => Medium ** Changed in: openldap (Ubuntu Noble) Importance: Undecided => Medium ** Changed in: openldap (Ubuntu Oracular) Importance: Undecided => Medium ** Changed in: openldap (Ubuntu Plucky) Importance: Undecided => Medium ** Changed in: openldap (Ubuntu Focal) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Changed in: openldap (Ubuntu Jammy) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Changed in: openldap (Ubuntu Noble) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Changed in: openldap (Ubuntu Oracular) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openldap in Ubuntu. https://bugs.launchpad.net/bugs/2090806 Title: TIMEOUT and NETWORK_TIMEOUT don't work when SSL is involved Status in openldap package in Ubuntu: In Progress Status in openldap source package in Focal: In Progress Status in openldap source package in Jammy: In Progress Status in openldap source package in Noble: In Progress Status in openldap source package in Oracular: In Progress Status in openldap source package in Plucky: In Progress Bug description: [Impact] If you connect to a slapd server using TLS and for some reason, the connection hangs or fails to completely establish, TIMEOUT and NETWORK_TIMEOUT values do not function as they are intended. They get ignored, and the connecting process e.g. ldapsearch just sits there at 100% CPU waiting forever. If slapd is set up in High Availability with servers to failover to, the failover will never occur, due to the connecting processes sitting there at 100% CPU waiting for the connection that will never succeed, leading to outages. You can simulate this easily with: $ ldapsearch -x -H ldaps://example.com $ sudo kill -STOP <slapd pid> $ ldapsearch -x -H ldaps://example.com and the last call will hang forever. There doesn't seem to be any workaround for this, apart from not using SSL. [Testcase] You can get away with just the one VM for this. Set up the ldap server (from server docs): https://ubuntu.com/server/docs/install-and-configure-ldap $ sudo apt install slapd ldap-utils $ sudo dpkg-reconfigure slapd Set DNS to example.com $ ldapsearch -x -LLL -H ldap:/// -b dc=example,dc=com dn dn: dc=example,dc=com $ vim add_content.ldif dn: ou=People,dc=example,dc=com objectClass: organizationalUnit ou: People dn: ou=Groups,dc=example,dc=com objectClass: organizationalUnit ou: Groups dn: cn=miners,ou=Groups,dc=example,dc=com objectClass: posixGroup cn: miners gidNumber: 5000 dn: uid=john,ou=People,dc=example,dc=com objectClass: inetOrgPerson objectClass: posixAccount objectClass: shadowAccount uid: john sn: Doe givenName: John cn: John Doe displayName: John Doe uidNumber: 10000 gidNumber: 5000 userPassword: {CRYPT}x gecos: John Doe loginShell: /bin/bash homeDirectory: /home/john $ ldapadd -x -D cn=admin,dc=example,dc=com -W -f add_content.ldif $ ldapsearch -x -LLL -b dc=example,dc=com '(uid=john)' cn gidNumber dn: uid=john,ou=People,dc=example,dc=com cn: John Doe gidNumber: 5000 $ ldappasswd -x -D cn=admin,dc=example,dc=com -W -S uid=john,ou=people,dc=example,dc=com New password: Re-enter new password: Enter LDAP Password: Configure the server for SSL (from server docs): https://ubuntu.com/server/docs/ldap-and-transport-layer-security-tls $ sudo apt install gnutls-bin ssl-cert $ sudo certtool --generate-privkey --bits 4096 --outfile /etc/ssl/private/mycakey.pem $ sudo vim /etc/ssl/ca.info cn = Example Company ca cert_signing_key expiration_days = 3650 $ sudo certtool --generate-self-signed \ --load-privkey /etc/ssl/private/mycakey.pem \ --template /etc/ssl/ca.info \ --outfile /usr/local/share/ca-certificates/mycacert.crt $ sudo update-ca-certificates $ sudo certtool --generate-privkey \ --bits 2048 \ --outfile /etc/ldap/ldap01_slapd_key.pem $ sudo vim /etc/ssl/ldap01.info organization = Example Company cn = example.com tls_www_server encryption_key signing_key expiration_days = 365 $ sudo certtool --generate-certificate \ --load-privkey /etc/ldap/ldap01_slapd_key.pem \ --load-ca-certificate /etc/ssl/certs/mycacert.pem \ --load-ca-privkey /etc/ssl/private/mycakey.pem \ --template /etc/ssl/ldap01.info \ --outfile /etc/ldap/ldap01_slapd_cert.pem $ sudo chgrp openldap /etc/ldap/ldap01_slapd_key.pem $ sudo chmod 0640 /etc/ldap/ldap01_slapd_key.pem $ sudo vim certinfo.ldif dn: cn=config add: olcTLSCACertificateFile olcTLSCACertificateFile: /etc/ssl/certs/mycacert.pem - add: olcTLSCertificateFile olcTLSCertificateFile: /etc/ldap/ldap01_slapd_cert.pem - add: olcTLSCertificateKeyFile olcTLSCertificateKeyFile: /etc/ldap/ldap01_slapd_key.pem $ sudo ldapmodify -Y EXTERNAL -H ldapi:/// -f certinfo.ldif $ sudo vim /etc/default/slapd Change LAPD_SERVICES="ldap:/// ldapi:///" to SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///" Next add TIMEOUT and NETWORK_TIMEOUT to the config file. $ sudo vim /etc/ldap/ldap.conf NETWORK_TIMEOUT 30 TIMEOUT 30 $ sudo systemctl restart slapd Reproduce the problem: $ ldapsearch -x -H ldaps://example.com # extended LDIF # # LDAPv3 # base <> (default) with scope subtree # filter: (objectclass=*) # requesting: ALL # # search result search: 2 result: 32 No such object # numResponses: 1 $ ps aux | grep slapd openldap 4097 0.0 0.1 1224136 7876 ? Ssl 02:22 0:00 /usr/sbin/slapd -h ldap:/// ldapi:/// ldaps:/// -g openldap -u openldap -F /etc/ldap/slapd.d $ sudo kill -STOP 4097 $ ldapsearch -x -H ldaps://example.com Check top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4237 ubuntu 20 0 20724 10956 6656 R 100.0 0.3 0:35.70 ldapsearch wait for 30 seconds, see nothing changes. If you install packages from the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf401397-test You should see the timeout work as expected, and the ldapsearch process will give up after 30 seconds. [Where problems could occur] We are changing how connections are tracked within openldap to better accurately say what state we are in, and for the timeouts to work when the connection doesn't make any progress. This will cause TIMEOUT and NETWORK_TIMEOUT to start working as intended, which might surprise some sysadmins who configured it, and likely forgot about it when they never had any timeout issues. If a regression were to occur, it would affect initial connections to openldap servers, or even long running commands that might exceed the TIMEOUT on queries over large ldap databases. [Other info] Upstream issues: https://bugs.openldap.org/show_bug.cgi?id=8047 https://bugs.openldap.org/show_bug.cgi?id=10141 This was fixed in 2.6.9 upstream by: commit d143f7a2dc82fb66e7741b93a1ae9e874ce2ac46 Author: Ondřej Kuzník <on...@mistotebe.net> Date: Mon, 21 Oct 2024 11:50:11 +0100 Subject: ITS#8047 Fix TLS connection timeout handling Link: https://git.openldap.org/openldap/openldap/-/commit/d143f7a2dc82fb66e7741b93a1ae9e874ce2ac46 This fix is required in all releases. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openldap/+bug/2090806/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
