This bug was fixed in the package systemd - 245.4-4ubuntu3.16
---------------
systemd (245.4-4ubuntu3.16) focal; urgency=medium
[ Dan Streetman ]
* d/p/lp1946388-sd-journal-don-t-check-namespaces-if-we-have-no-name.patch:
Avoid journalctl segfault (LP: #1946388)
[ Jeremy Szu ]
* Add a allowlist to unblock intel-hid on new HP machines (LP: #1955997)
Author: Jeremy Szu
File:
debian/patches/lp1955997-add-a-allowlist-to-unblock-intel-hid-on-HP-mach.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=88a859eaddb6c9a611fcbc44edab441aef4c4355
[ Nick Rosbrook ]
* Prevent arguments from being overwritten with defaults at shutdown (LP:
#1958284)
File:
debian/patches/lp1958284-core-move-reset_arguments-to-the-end-of-main-s-finish.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e61052bd1f20bcc54e7417542c6d445cf5040f56
[ Lukas Märdian ]
* Fix deadlock between pid1 and dbus-daemon (LP: #1871538)
Author: Lukas Märdian
File:
debian/patches/pid1-set-SYSTEMD_NSS_DYNAMIC_BYPASS-1-env-var-for-dbus-da.patch
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e3aacfa26e3fc6df369e6f28e740389ae0020907
-- Nick Rosbrook <nick.rosbr...@canonical.com> Wed, 23 Mar 2022
09:29:33 -0400
** Changed in: systemd (Ubuntu Focal)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1958284
Title:
shutdown hangs at "Waiting for process: ..." for 90s, ignoring
DefaultTimeoutStopSec
Status in systemd package in Ubuntu:
Fix Released
Status in systemd source package in Focal:
Fix Released
Bug description:
[Impact]
The systemd shutdown sequence does not honor systemd-system.conf
settings when waiting for remaining processes. This means that, for
example, if a systemd service specifies KillMode=process and a process
remaining from that service does not properly handle SIGTERM, then the
remaining process will not be killed until after the compiled-in
default value of DefaultTimeoutStopSec (90s), even if the user has
changed the setting of DefaultTimeoutStopSec. In such cases, this
impacts users by significantly increasing the time required for
shutdown/reboot.
[Test Plan]
* Create a new script, /usr/local/bin/loop-ignore-sigterm:
```
#!/bin/bash
loop_forever() {
while true; do sleep 1; done
}
(
trap 'echo Ignoring SIGTERM...' SIGTERM
loop_forever
)
loop_forever
```
This script will spawn a subshell which will loop forever and ignore
SIGTERM. This will force systemd to wait for the subprocess at
reboot/shutdown, and eventually send SIGKILL after TimeoutStopSec
(DefaultTimeoutStopSec in this case).
* Make the script executable:
$ chmod +x /usr/local/bin/loop-ignore-sigterm
* Create a systemd service for this script. Add the following to
/etc/systemd/system/loop-ignore-sigterm.service:
```
[Service]
KillMode=process
ExecStart=/usr/local/bin/loop-ignore-sigterm
```
* Start the service:
$ systemctl start loop-ignore-sigterm.service
* Edit /etc/systemd/system.conf, and uncomment the
'DefaultTimeoutStopSec=90s' line. Modify 90s to something much shorter,
e.g. 20s.
* Re-exec the daemon so this new default takes effect:
$ systemctl daemon-reexec
* Reboot, and monitor the logs. Observe that systemd-shutdown will wait
for the loop-ignore-sigterm process for 90s, instead of the 20s
configured earlier.
[Where problems could occur]
The patch moves the reset_arguments() call to the end of main, which
means reset_arguments() is no longer called before daemon re-execution
(if that branch is taken). If anything in that code path relied on
reset_arguments() being called before re-executing, those assumptions
could be broken. Any such problems would potentially be seen during
daemon re-execution, e.g. when calling systemctl daemon-reexec.
[ Original Description ]
With systemd v245 as shipped with 20.04, the shutdown sequence does
not use the value of `DefaultTimeoutStopSec` to wait for remaining
processes, it instead uses the compiled in default of 90s.
This is most visible with services that use `KillMode=process`
(docker, k8s, k3s, etc...), especially if the remaining processes do
not handle `SIGTERM` or choose to ignore it.
For example:
```
[ OK ] Finished Reboot.
[ OK ] Reached target Reboot.
[ 243.652848 ] systemd-shutdown[1]: Waiting for process: containerd-shim,
containerd-shim, containerd-shim, fluent-bit
--- hangs here for 90s even if DefaultTimeoutStopSec is set to a lower
value ---
```
The bug has been fixed upstream here:
https://github.com/systemd/systemd/commit/7d9eea2bd3d4f83668c7a78754d201b22
Marc was kind enough to package the patch for 20.04 so I could test it
(https://launchpad.net/~mdeslaur/+archive/ubuntu/testing/+sourcepub/13210617/+listing-
archive-extra) and with that package, I can confirm that it indeed
fixes the issue.
Here's a few github issues I stumbled upon while trying to debug this,
along with a short writeup of the workaround I ended up using:
- https://github.com/moby/moby/issues/41831
- https://github.com/k3s-io/k3s/issues/2400
- https://github.com/systemd/systemd/issues/16991
- https://raby.sh/debugging-90s-hangs-during-shutdown-on-ubuntu-2004.html
Of course, it would be much better if all the processes would properly
handle `SIGTERM`, but having a way to enforce a maximum wait time at
shutdown is a decent workaround.
Given that the patch is relatively simple, would it be possible to add
it the package for 20.04?
Thanks
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1958284/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
