This bug was fixed in the package unattended-upgrades -
1.5ubuntu3.18.10.0
---------------
unattended-upgrades (1.5ubuntu3.18.10.0) cosmic; urgency=medium
* Revert to running unattended-upgrades.service in multi-user.target
* Trigger unattended-upgrade-shutdown actions with PrepareForShutdown()
Performing upgrades in service's ExecStop did not work when the upgrades
involved restarting services because systemd blocked other stop/start
actions making maintainer scripts time out and be killed leaving a broken
system behind.
Running unattended-upgrades.service before shutdown.target as a oneshot
service made it run after unmounting filesystems and scheduling services
properly on shutdown is a complex problem and adding more services to the
mix make it even more fragile.
The solution of monitoring PrepareForShutdown() signal from DBus
allows Unattended Upgrade to run _before_ the jobs related to shutdown are
queued thus package upgrades can safely restart services without
risking causing deadlocks or breaking part of the shutdown actions.
Also ask running unattended-upgrades to stop when shutdown starts even in
InstallOnShutdown mode and refactor most of unattended-upgrade-shutdown to
UnattendedUpgradesShutdown class. (LP: #1778219, LP: #1803137)
* Handle reverting to WantedBy=multi-user.target
* Increase logind's InhibitDelayMaxSec to 30s.
This allows more time for unattended-upgrades to shut down gracefully
or even install a few packages in InstallOnShutdown mode, but is still a
big step back from the 30 minutes allowed for InstallOnShutdown previously.
Users enabling InstallOnShutdown mode are advised to increase
InhibitDelayMaxSec even further possibly to 30 minutes.
* Cache polling result for PreparingForShutdown after it becomes true
* debian/tests/test-systemd.py: Reboot system with dbus call to honor
inhibitor locks
* Add NEWS entry about increasing InhibitDelayMaxSec and InstallOnShutdown
changes
* Stop using ActionGroups, they interfere with apt.Cache.clear()
causing all autoremovable packages to be handled as newly autoremovable ones
and be removed by default. Dropping ActionGroup usage does not slow down the
most frequent case of not having anything to upgrade and when ther are
packages to upgrade the gain is small compared to the actual package
installation.
Also collect autoremovable packages before adjusting candidates because that
also changed .is_auto_removable attribute of some of them. (LP: #1803749)
(Closes: #910874)
-- Balint Reczey <rbal...@ubuntu.com> Mon, 26 Nov 2018 12:28:55 +0100
** Changed in: unattended-upgrades (Ubuntu Cosmic)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1778219
Title:
unattended-upgrades hangs on shutdown, leaves system in a broken state
Status in apt package in Ubuntu:
Invalid
Status in init-system-helpers package in Ubuntu:
Invalid
Status in snapd package in Ubuntu:
Confirmed
Status in unattended-upgrades package in Ubuntu:
Fix Released
Status in apt source package in Xenial:
Confirmed
Status in init-system-helpers source package in Xenial:
Confirmed
Status in snapd source package in Xenial:
Confirmed
Status in unattended-upgrades source package in Xenial:
In Progress
Status in apt source package in Bionic:
Confirmed
Status in init-system-helpers source package in Bionic:
Confirmed
Status in snapd source package in Bionic:
Confirmed
Status in unattended-upgrades source package in Bionic:
Fix Released
Status in unattended-upgrades source package in Cosmic:
Fix Released
Bug description:
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades
that stat/stop services on shutdown, leaving the system in a broken
state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking
that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' >
/etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins::
"${distro_id}:${distro_codename}-updates";' >
/etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1
/org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system
rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600
please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly
transition from coupling u-u.service with multi-user.target to coupling it with
shutdown.target again which can make u-u started during normal boot when there
is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and
targets u-u may fail to run on shutdown in case of an unexpected regression.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the
package installation on various packages hangs until the systemd
ShutdownTimeout (30min) is expired and systemd kills all processes and
powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least
sometimes this cannot be fixed with a "dpkg --configure -a", but
instead requires the user to manually reinstall the package that
caused the hang.
This appears to be a deadlock, because the hanging commands are always
"systemctl stop ..." or "systemctl restart ...", etc.. If I understand
this correctly, those systemctl commands block because systemd tries
to shutdown the system and tries to satisfy all dependencies for the
shutdown targets before those systemctl commands could get executed,
which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package.
The hanging command is "systemctl stop apport-forward.socket". The
system hangs until the systemd ShutdownTimeout expires and systemd
forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and
needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the
default configuration unattended-updates does not use bionic-updates,
and seems to have skipped installation of apport from bionic-security
(supposedly because an already newer version of apport was in bionic-
updates). If my understanding of why apport initially did not get
installed is correct, then this would be another problem, because it
would mean that unattended-upgrades potentially does not install all
available security updates when bionic-updates is enabled in
sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in
sources.list, but instead enabling bionic-updates in
/etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did
hang when installing the package snapd (the hanging command was "systemctl stop
snapd.autoimport.service snapd.core-fixup.service snapd.service
snapd.snap-repair.service snapd.snap-repair.service snapd.socket
snapd.system-shutdown.service"). This leads to the same problems as described
above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1778219/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
