[systemd-devel] systemctl daemon-reexec forgets running services and starts everything new
Hello List, I'm posting this here because the official bugtracker at https://github.com/systemd/systemd/issues accepts only reports for new versions of systemd. This is a bug report for version 249. # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd-sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. The last working version is version `245.4-4ubuntu3.19`. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found left-over process 164 (cron) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Regular background program processing daemon. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: dbus.service: Found left-over process 177 (dbus-daemon) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started D-Bus System Message Bus. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: rsyslog.service: Found left-over process 204 (rsyslogd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Failed with result 'exit-code'. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Unit process 206 (atd) remains running after unit stopped. Mar 31 12:51:39 FQDN_REDACTED systemd[1
Re: [systemd-devel] systemctl daemon-reexec forgets running services and starts everything new
> On 4 Apr 2023, at 09:33, Wasser, Erik wrote: > > Hello List, > > I'm posting this here because the official bugtracker at > https://github.com/systemd/systemd/issues accepts only reports for new > versions of systemd. This is a bug report for version 249. > > # Our problem # > > During a regular update of our container environment, `systemd` (and > the related packages libpam-systemd, libsystemd0, libudev1, > systemd-sysv and udev) were updated from `249.11-0ubuntu3.6` to > `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu > 20.04 is working fine with `systemctl daemon-reexec`. > > In my opinion, the update was not the problem because we've tried > downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, > `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The > symptoms were the same. The last working version is version > `245.4-4ubuntu3.19`. > > # Symptoms # > > The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl > daemon-reexec` and that ended in a disaster. Seems it is a packaging issue that ubuntu or debian needs to help you with not systemd? Barry > It seems that `systemd` > is forgetting all it started children and tries to start nearly every > configured service again. Naturally, the old services are still > running, and the ports can't be opened twice and `systemd` won't give > up. Here are some(!) of the logfiles: > > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile > Files and Directories... > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found > left-over process 130 (systemd-udevd) in control group while starting > unit. Ignoring. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates > unclean termination of a previous run, or service implementation > deficiencies. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found > left-over process 31475 (systemd-udevd) in control group while > starting unit. Ignoring. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates > unclean termination of a previous run, or service implementation > deficiencies. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found > left-over process 31476 (systemd-udevd) in control group while > starting unit. Ignoring. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates > unclean termination of a previous run, or service implementation > deficiencies. > > And... > > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System > Initialization. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download > activities. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade > and clean activities. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database > backup timer. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online > Metadata Check for All Filesystems. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in > Discard unused blocks once a week being skipped. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session > files every 30 mins. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate > database daily. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of > Temporary Directories. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. > > And... > > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over > process 206 (atd) in control group while starting unit. Ignoring. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates > unclean termination of a previous run, or service implementation > deficiencies. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution > scheduler... > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found > left-over process 164 (cron) in control group while starting unit. > Ignoring. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates > unclean termination of a previous run, or service implementation > deficiencies. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Regular background > program processing daemon. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: dbus.service: Found > left-over process 177 (dbus-daemon) in control group while starting > unit. Ignoring. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates > unclean termination of a previous run, or service implementation > deficiencies. > Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started D-Bus System Message Bus. > > And... > > Mar 31 12:51:39 FQD
