Thank you Michael for filing this bug and improving cloud-init. It appears debhelper postinstall scripts injected into cloud-init package builds didn't support a directory change in our packaging and the resulting binary deb never enabled cloud-init services. We'll back out that directory change on Jammy. This doesn't appear to affect Noble or newer.
In cloud-init 25.1.2, changed systemd-related packaging from /lib TO /usr/lib which was suggested by lintian packaging warnings.[1] This change though is incompatible with debhelper version 13.6ubuntu1 in Jammy which now no longer "sees" cloud-init's systemd units/services/target and doesn't inject the same debhelper enablement script segments for all of cloud-init services. This results in inability to launch any VMs on Jammy as cloud-init services are all disabled/insactive despite ds-indentify correctly identifying the platform. This behavior was seen in MAAS Jammy images being unable to get to network with cloud-init 25.1.2 per this comment[2]. on a somewhat related MAAS bug about early boot on Jammy. This regression was caused by the single commit: https://github.com/canonical/cloud-init/commit/0547349214fcfb827e58c1de5e4ad7d23d08cc7f With the above change, the binary installed by cloud-init 25.1.2 will not enable any cloud-init services because cloud-init.postinst lacks all the following debhelper enablement scripts: # Automatically added by dh_installsystemd/13.6ubuntu1 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then # This will only remove masks created by d-s-h on package removal. deb-systemd-helper unmask 'cloud-config.service' >/dev/null || true # was-enabled defaults to true, so new installations run enable. if deb-systemd-helper --quiet was-enabled 'cloud-config.service'; then # Enables the unit on first installation, creates new # symlinks on upgrades if the unit file has changed. deb-systemd-helper enable 'cloud-config.service' >/dev/null || true else # Update the statefile to add new symlinks (if any), which need to be # cleaned up on purge. Also remove old symlinks. deb-systemd-helper update-state 'cloud-config.service' >/dev/null || true fi fi # End automatically added section # Automatically added by dh_installsystemd/13.6ubuntu1 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then # This will only remove masks created by d-s-h on package removal. deb-systemd-helper unmask 'cloud-final.service' >/dev/null || true # was-enabled defaults to true, so new installations run enable. if deb-systemd-helper --quiet was-enabled 'cloud-final.service'; then # Enables the unit on first installation, creates new # symlinks on upgrades if the unit file has changed. deb-systemd-helper enable 'cloud-final.service' >/dev/null || true else # Update the statefile to add new symlinks (if any), which need to be # cleaned up on purge. Also remove old symlinks. deb-systemd-helper update-state 'cloud-final.service' >/dev/null || true fi fi # End automatically added section # Automatically added by dh_installsystemd/13.6ubuntu1 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then # This will only remove masks created by d-s-h on package removal. deb-systemd-helper unmask 'cloud-init-hotplugd.socket' >/dev/null || true # was-enabled defaults to true, so new installations run enable. if deb-systemd-helper --quiet was-enabled 'cloud-init-hotplugd.socket'; then # Enables the unit on first installation, creates new # symlinks on upgrades if the unit file has changed. deb-systemd-helper enable 'cloud-init-hotplugd.socket' >/dev/null || true else # Update the statefile to add new symlinks (if any), which need to be # cleaned up on purge. Also remove old symlinks. deb-systemd-helper update-state 'cloud-init-hotplugd.socket' >/dev/null || true fi fi # End automatically added section # Automatically added by dh_installsystemd/13.6ubuntu1 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then # This will only remove masks created by d-s-h on package removal. deb-systemd-helper unmask 'cloud-init-local.service' >/dev/null || true # was-enabled defaults to true, so new installations run enable. if deb-systemd-helper --quiet was-enabled 'cloud-init-local.service'; then # Enables the unit on first installation, creates new # symlinks on upgrades if the unit file has changed. deb-systemd-helper enable 'cloud-init-local.service' >/dev/null || true else # Update the statefile to add new symlinks (if any), which need to be # cleaned up on purge. Also remove old symlinks. deb-systemd-helper update-state 'cloud-init-local.service' >/dev/null || true fi fi # End automatically added section # Automatically added by dh_installsystemd/13.6ubuntu1 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then # This will only remove masks created by d-s-h on package removal. deb-systemd-helper unmask 'cloud-init.service' >/dev/null || true # was-enabled defaults to true, so new installations run enable. if deb-systemd-helper --quiet was-enabled 'cloud-init.service'; then # Enables the unit on first installation, creates new # symlinks on upgrades if the unit file has changed. deb-systemd-helper enable 'cloud-init.service' >/dev/null || true else # Update the statefile to add new symlinks (if any), which need to be # cleaned up on purge. Also remove old symlinks. deb-systemd-helper update-state 'cloud-init.service' >/dev/null || true fi fi # End automatically added section References: [1] Upstream cloud-init change doc https://cloudinit.readthedocs.io/en/latest/reference/breaking_changes.html#id2 [2] MAAS image launch failures https://bugs.launchpad.net/maas/+bug/2106671/comments/21 ** Also affects: cloud-init (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: cloud-init (Ubuntu) Importance: Undecided => Critical ** Changed in: cloud-init (Ubuntu Jammy) Status: New => In Progress ** Description changed: + In cloud-init 25.1.2, changed systemd-related packaging from /lib TO /usr/lib + which was suggested by lintian packaging warnings.[1] + + This change though is incompatible with debhelper version 13.6ubuntu1 in + Jammy which now no longer "sees" cloud-init's systemd + units/services/target and doesn't inject the same debhelper enablement + script segments for all of cloud-init services. This results in + inability to launch any VMs on Jammy as cloud-init services are all + disabled/insactive despite ds-indentify correctly identifying the + platform. + + This behavior was seen in MAAS Jammy images being unable to get to + network with cloud-init 25.1.2 per this comment[2]. on a somewhat + related MAAS bug about early boot on Jammy. + + This regression was caused by the single commit: + https://github.com/canonical/cloud-init/commit/0547349214fcfb827e58c1de5e4ad7d23d08cc7f + + With the above change, the binary installed by cloud-init 25.1.2 will + not enable any cloud-init services because cloud-init.postinst lacks all + the following debhelper enablement scripts: + + # Automatically added by dh_installsystemd/13.6ubuntu1 + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + # This will only remove masks created by d-s-h on package removal. + deb-systemd-helper unmask 'cloud-config.service' >/dev/null || true + + # was-enabled defaults to true, so new installations run enable. + if deb-systemd-helper --quiet was-enabled 'cloud-config.service'; then + # Enables the unit on first installation, creates new + # symlinks on upgrades if the unit file has changed. + deb-systemd-helper enable 'cloud-config.service' >/dev/null || true + else + # Update the statefile to add new symlinks (if any), which need to be + # cleaned up on purge. Also remove old symlinks. + deb-systemd-helper update-state 'cloud-config.service' >/dev/null || true + fi + fi + # End automatically added section + # Automatically added by dh_installsystemd/13.6ubuntu1 + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + # This will only remove masks created by d-s-h on package removal. + deb-systemd-helper unmask 'cloud-final.service' >/dev/null || true + + # was-enabled defaults to true, so new installations run enable. + if deb-systemd-helper --quiet was-enabled 'cloud-final.service'; then + # Enables the unit on first installation, creates new + # symlinks on upgrades if the unit file has changed. + deb-systemd-helper enable 'cloud-final.service' >/dev/null || true + else + # Update the statefile to add new symlinks (if any), which need to be + # cleaned up on purge. Also remove old symlinks. + deb-systemd-helper update-state 'cloud-final.service' >/dev/null || true + fi + fi + # End automatically added section + # Automatically added by dh_installsystemd/13.6ubuntu1 + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + # This will only remove masks created by d-s-h on package removal. + deb-systemd-helper unmask 'cloud-init-hotplugd.socket' >/dev/null || true + + # was-enabled defaults to true, so new installations run enable. + if deb-systemd-helper --quiet was-enabled 'cloud-init-hotplugd.socket'; then + # Enables the unit on first installation, creates new + # symlinks on upgrades if the unit file has changed. + deb-systemd-helper enable 'cloud-init-hotplugd.socket' >/dev/null || true + else + # Update the statefile to add new symlinks (if any), which need to be + # cleaned up on purge. Also remove old symlinks. + deb-systemd-helper update-state 'cloud-init-hotplugd.socket' >/dev/null || true + fi + fi + # End automatically added section + # Automatically added by dh_installsystemd/13.6ubuntu1 + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + # This will only remove masks created by d-s-h on package removal. + deb-systemd-helper unmask 'cloud-init-local.service' >/dev/null || true + + # was-enabled defaults to true, so new installations run enable. + if deb-systemd-helper --quiet was-enabled 'cloud-init-local.service'; then + # Enables the unit on first installation, creates new + # symlinks on upgrades if the unit file has changed. + deb-systemd-helper enable 'cloud-init-local.service' >/dev/null || true + else + # Update the statefile to add new symlinks (if any), which need to be + # cleaned up on purge. Also remove old symlinks. + deb-systemd-helper update-state 'cloud-init-local.service' >/dev/null || true + fi + fi + # End automatically added section + # Automatically added by dh_installsystemd/13.6ubuntu1 + if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then + # This will only remove masks created by d-s-h on package removal. + deb-systemd-helper unmask 'cloud-init.service' >/dev/null || true + + # was-enabled defaults to true, so new installations run enable. + if deb-systemd-helper --quiet was-enabled 'cloud-init.service'; then + # Enables the unit on first installation, creates new + # symlinks on upgrades if the unit file has changed. + deb-systemd-helper enable 'cloud-init.service' >/dev/null || true + else + # Update the statefile to add new symlinks (if any), which need to be + # cleaned up on purge. Also remove old symlinks. + deb-systemd-helper update-state 'cloud-init.service' >/dev/null || true + fi + fi + # End automatically added section + + References: + + [1] Upstream cloud-init change doc + https://cloudinit.readthedocs.io/en/latest/reference/breaking_changes.html#id2 + + [2] MAAS image launch failures + https://bugs.launchpad.net/maas/+bug/2106671/comments/21 + + + + + =============== Original description ================ + DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS" apt-cache policy cloud-init cloud-init: - Installed: 25.1.2-0ubuntu0~22.04.1 - Candidate: 25.1.2-0ubuntu0~22.04.1 - Version table: - *** 25.1.2-0ubuntu0~22.04.1 100 - 100 /var/lib/dpkg/status + Installed: 25.1.2-0ubuntu0~22.04.1 + Candidate: 25.1.2-0ubuntu0~22.04.1 + Version table: + *** 25.1.2-0ubuntu0~22.04.1 100 + 100 /var/lib/dpkg/status At some point after June 4th 2025 our Octavia OpenStack test jobs started to fail. On closer inspection we found that the VMs were not getting IP addresses configured. I was able to reproduce this on a local devstack system. On the failing VM I connected to the console locally. The cloud-init configuration appears correct and the config drive mounted with the correct information. What is odd however is there are no cloud-init logs in /var/log. It appears as if it didn't run on boot. I compared our images from the last known successful test run and the current image and the only package I see different is: < cloud-init all 24.4.1-0ubuntu0~22.04.2 [566 kB] --- > cloud-init all 25.1.2-0ubuntu0~22.04.1 [568 kB] I then made a copy of the image and replaced 25.1.2 with the 24.4.1 version and this image booted as expected. This issue is 100% repeatable with 25.1.2. If you would like to create an image we have tools available to create one that reproduces the problem: 1. git clone https://opendev.org/openstack/octavia 2. cd octavia/diskimage-create 3. run "./diskimage-create.sh -r <root password> -d jammy" 4. It will create a qcow2 image that boots under OpenStack with config drive ** Summary changed: - cloud-init 25.1.2 on Jammy not running on VM boot + cloud-init 25.1.2 on Jammy not running on VM boot, systemd services disabled -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2113797 Title: cloud-init 25.1.2 on Jammy not running on VM boot, systemd services disabled To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/2113797/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
