** Description changed: [Impact] This release sports both bug-fixes and new features and we would like to make sure all of our supported customers have access to these improvements. The notable ones are: * more robust error handling when determining the cloud we're on LP: #1940131 LP: #1938207 LP: #1944676 * disallows fips on focal aws/azure LP: #1939449 LP: #1939932 * adds/changes to ua-related recurring jobs: - change in frequency to existing job: updates the apt and motd esm update messaging: every 6 hours - new job: updates the contract details and status: every 12 hours - new job: ONLY ON GCP (implemented as separate timer that is only activated on GCP LTS when not attached): checks for license changes and auto-attaches if a pro license was added: every 5 minutes * adds support for ros/ros-updates entitlements with --beta flag With this change, the ua-message timer is renamed to ua-timer, as it has a more generic functionality: it triggers sub jobs which need to be executed periodically. One of those is exactly the job which updates the messaging - which has its interval reduced to 6h, as well as the timer itself. There is also a job to update the client status every 12h, and a third one to collect metrics, which is disabled for this release. See the changelog entry below for a full list of changes and bugs. [Test Case] The following development and SRU process was followed: https://wiki.ubuntu.com/UbuntuAdvantageToolsUpdates The ubuntu-advantage-tools team will be in charge of attaching the artifacts and console output of the appropriate run to the bug. ubuntu-advantage-tools team members will not mark ‘verification-done’ until this has happened. <TODO After released to -proposed: attach integration test artifacts> In addition to the automated integration testing, here are manual test instructions to verify that the packaging changes are functional. All of these tests use the ua-client/staging PPA, which has the 27.3 release build in it. Manual Test 1: - Here we verify the cloud-id command changes in postinst work correct such that installation succeeds even if the cloud-id command fails - ``` - lxc launch ubuntu-daily:impish dev-i - lxc exec dev-i -- sh -c "printf 'exit 1' > /usr/bin/cloud-id" - lxc exec dev-i chmod +x /usr/bin/cloud-id - lxc exec dev-i cloud-id + Here we verify that the cloud-id command changes in postinst work correctly such that the installation succeeds even if the cloud-id command fails. + ``` + lxc launch ubuntu-daily:impish dev-i + lxc exec dev-i -- /bin/sh -c "printf 'exit 1' > /usr/bin/cloud-id" + lxc exec dev-i -- chmod +x /usr/bin/cloud-id + lxc exec dev-i -- cloud-id # CHECK: successfully faking cloud-init error. command should've exited 1 - lxc exec dev-i add-apt-repository ppa:ua-client/staging - lxc exec dev-i apt install ubuntu-advantage-tools + lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-i -- apt install -y ubuntu-advantage-tools # CHECK: installation should've succeeded despite failing cloud-init lxc delete dev-i --force ``` Manual Test 2: Here we verify that the new license check timer only runs on LTS instances that are identified as running on GCP. ``` # should not be enabled when not gcp, on LTS (focal) lxc launch ubuntu-daily:focal dev-f - lxc exec dev-f add-apt-repository ppa:ua-client/staging - lxc exec dev-f apt install ubuntu-advantage-tools + lxc exec dev-f -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-f -- apt install -y ubuntu-advantage-tools lxc exec dev-f -- systemctl list-timers --all - lxc exec dev-f systemctl status ua-license-check.timer + lxc exec dev-f -- systemctl status ua-license-check.timer # CHECK: should not be running - lxc exec dev-f reboot + lxc exec dev-f -- reboot lxc exec dev-f -- systemctl list-timers --all - lxc exec dev-f systemctl status ua-license-check.timer + lxc exec dev-f -- systemctl status ua-license-check.timer # CHECK: still should not be running lxc delete dev-f --force # fake gcp by overwriting cloud-id, still should not be enabled because not on LTS (impish) lxc launch ubuntu-daily:impish dev-i - lxc exec dev-i -- sh -c "printf 'echo gce' > /usr/bin/cloud-id" - lxc exec dev-i chmod +x /usr/bin/cloud-id - lxc exec dev-i cloud-id + lxc exec dev-i -- /bin/sh -c "printf 'echo gce' > /usr/bin/cloud-id" + lxc exec dev-i -- chmod +x /usr/bin/cloud-id + lxc exec dev-i -- cloud-id # CHECK: successfully faking gcp. output should be "gce" (with an "e") - lxc exec dev-i add-apt-repository ppa:ua-client/staging - lxc exec dev-i apt install ubuntu-advantage-tools - lxc exec dev-i -- systemctl list-timers --all - lxc exec dev-i systemctl status ua-license-check.timer + lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-i -- apt install -y ubuntu-advantage-tools + lxc exec dev-i -- systemctl list-timers --all + lxc exec dev-i -- systemctl status ua-license-check.timer # CHECK: should not be running lxc delete dev-i --force # fake gcp by overwriting cloud-id, on LTS (focal), should be enabled lxc launch ubuntu-daily:focal dev-f - lxc exec dev-f -- sh -c "printf 'echo gce' > /usr/bin/cloud-id" - lxc exec dev-f chmod +x /usr/bin/cloud-id - lxc exec dev-f cloud-id + lxc exec dev-f -- /bin/sh -c "printf 'echo gce' > /usr/bin/cloud-id" + lxc exec dev-f -- chmod +x /usr/bin/cloud-id + lxc exec dev-f -- cloud-id # CHECK: successfully faking gcp. output should be "gce" (with an "e") - lxc exec dev-f add-apt-repository ppa:ua-client/staging - lxc exec dev-f apt install ubuntu-advantage-tools + lxc exec dev-f -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-f -- apt install -y ubuntu-advantage-tools lxc exec dev-f -- systemctl list-timers --all - lxc exec dev-f systemctl status ua-license-check.timer + lxc exec dev-f -- systemctl status ua-license-check.timer # CHECK: should be enabled lxc delete dev-f --force ``` Manual Test 3: - Here we verify that the old ua-messaging.timer,service artifacts are properly removed on upgrade - ``` - lxc launch ubuntu-daily:impish dev-i - lxc exec dev-i -- sh -c "ls -1 /etc/systemd/system/timers.target.wants /usr/lib/systemd/system /usr/lib/systemd/system /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants /var/lib/systemd/deb-systemd-helper-enabled/ | grep ua-messaging" + Here we verify that the old ua-messaging.{timer,service} artifacts are properly removed on upgrade. + ``` + lxc launch ubuntu-daily:impish dev-i + lxc exec dev-i -- /bin/sh -c "ls -1 /etc/systemd/system/timers.target.wants /usr/lib/systemd/system /usr/lib/systemd/system /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants /var/lib/systemd/deb-systemd-helper-enabled/ | grep ua-messaging" # CHECK: verify several ua-messaging artifacts - lxc exec dev-i add-apt-repository ppa:ua-client/staging - lxc exec dev-i apt install ubuntu-advantage-tools - lxc exec dev-i -- sh -c "ls -1 /etc/systemd/system/timers.target.wants /usr/lib/systemd/system /usr/lib/systemd/system /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants /var/lib/systemd/deb-systemd-helper-enabled/ | grep ua-messaging" + lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-i -- apt install -y ubuntu-advantage-tools + lxc exec dev-i -- /bin/sh -c "ls -1 /etc/systemd/system/timers.target.wants /usr/lib/systemd/system /usr/lib/systemd/system /var/lib/systemd/deb-systemd-helper-enabled/timers.target.wants /var/lib/systemd/deb-systemd-helper-enabled/ | grep ua-messaging" # CHECK: verify ua-messaging artifacts are not left behind lxc delete dev-i --force ``` Manual Test 4: Here we verify that if the user had disabled the old ua-messaging timer, then we will carry that preference forward to the new ua-timer timer. ``` lxc launch ubuntu-daily:impish dev-i lxc exec dev-i -- systemctl list-timers --all - lxc exec dev-i systemctl status ua-messaging.timer + lxc exec dev-i -- systemctl status ua-messaging.timer # CHECK: verify ua-messaging.timer is enabled - lxc exec dev-i systemctl stop ua-messaging.timer - lxc exec dev-i systemctl disable ua-messaging.timer - lxc exec dev-i -- systemctl list-timers --all - lxc exec dev-i systemctl status ua-messaging.timer + lxc exec dev-i -- systemctl stop ua-messaging.timer + lxc exec dev-i -- systemctl disable ua-messaging.timer + lxc exec dev-i -- systemctl list-timers --all + lxc exec dev-i -- systemctl status ua-messaging.timer # CHECK: verify ua-messaging.timer is disabled - lxc exec dev-i add-apt-repository ppa:ua-client/staging - lxc exec dev-i apt install ubuntu-advantage-tools - lxc exec dev-i -- systemctl list-timers --all - lxc exec dev-i systemctl status ua-timer.timer + lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-i -- apt install -y ubuntu-advantage-tools + lxc exec dev-i -- systemctl list-timers --all + lxc exec dev-i -- systemctl status ua-timer.timer # CHECK: verify ua-timer.timer is disabled lxc delete dev-i --force ``` Manual Test 5: Here we verify that the new log files are appropriately created on install, rotated by logrotate, and deleted on purge. ``` lxc launch ubuntu-daily:impish dev-i - lxc exec dev-i add-apt-repository ppa:ua-client/staging - lxc exec dev-i apt install ubuntu-advantage-tools - lxc exec dev-i -- sh -c "ls -l /var/log/ubuntu-advantage*" + lxc exec dev-i -- add-apt-repository -yu ppa:ua-client/staging + lxc exec dev-i -- apt install -y ubuntu-advantage-tools + lxc exec dev-i -- /bin/sh -c 'ls -l /var/log/ubuntu-advantage*' # CHECK: verify that three ua log files were created, are owned by root, and have 600 permissions - lxc exec dev-i -- sh -c "printf testcontent > /var/log/ubuntu-advantage.log" - lxc exec dev-i -- sh -c "printf testcontent > /var/log/ubuntu-advantage-timer.log" - lxc exec dev-i -- sh -c "printf testcontent > /var/log/ubuntu-advantage-license-check.log" + lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage.log" + lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-timer.log" + lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-license-check.log" lxc exec dev-i -- logrotate --force /etc/logrotate.d/ubuntu-advantage-tools - lxc exec dev-i -- sh -c "ls -l /var/log/ubuntu-advantage*" + lxc exec dev-i -- /bin/sh -c 'ls -l /var/log/ubuntu-advantage*' # CHECK: verify all 3 logs were rotated - lxc exec dev-i -- sh -c "printf testcontent > /var/log/ubuntu-advantage.log" - lxc exec dev-i -- sh -c "printf testcontent > /var/log/ubuntu-advantage-timer.log" - lxc exec dev-i -- sh -c "printf testcontent > /var/log/ubuntu-advantage-license-check.log" - lxc exec dev-i -- sh -c "ls /var/log/ubuntu-advantage*" + lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage.log" + lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-timer.log" + lxc exec dev-i -- /bin/sh -c "printf testcontent > /var/log/ubuntu-advantage-license-check.log" + lxc exec dev-i -- /bin/sh -c 'ls /var/log/ubuntu-advantage*' # CHECK: verify all ua log files exist including rotated versions - lxc exec dev-i apt purge ubuntu-advantage-tools - lxc exec dev-i -- sh -c "ls /var/log/" + lxc exec dev-i -- apt purge -y ubuntu-advantage-tools + lxc exec dev-i -- /bin/sh -c "ls /var/log/" # CHECK: verify that all ua log files are removed lxc delete dev-i --force ``` [Regression Potential] In order to mitigate the regression potential, the results of the aforementioned integration tests are attached to this bug. We moved the trigger of the apt and motd messaging updates from a dedicated systemd timer to a shared timer that conditionally calls the messaging updates in our python code. This adds complexity. If we made a mistake, then either the job won't get called frequently enough or will get called too frequently. If the former, then some esm updates related messaging will be out of date in apt and motd. If the latter, then cpu cycles will be wasted in needlessly updating messages. We touched postinst to handle cloud-id failures more robustly. Touching postinst is always scary because it is the most likely way for us to break upgrades. In theory this change made upgrades less likely to fail, but if we made a mistake, it could cause new unexpected failures. We added more recurring jobs in the service of new features. This increases complexity and potential for mistakes. In particular, we have strived to avoid excessive logging from these jobs. If we made a mistake in our logging, we could inadvertently fill up disks with useless logs. Additional recurring jobs will also use more cpu over time than previous versions. This is at least partially addressed below. We instrumented a high frequency timer to only run on GCP, but if we made a mistake, this could be accidentally activated on non-GCP machines, which would be a waste. (See below for additional high frequency timer discussion). We check if the ua-messaging timer was disabled prior to this update, and if so we also disable the new ua-timer timer in systemd. Failing to do so would keep enabled a service that the user had explicitly disabled in the past, resulting in a unwanted behavior. Our migration of this user configuration only covers the case where the user ran `systemctl disable` to disable the old timer. If they disabled the timer in a different way, then their configuration will not be carried forward. Furthermore, this is a somewhat complicated postinst addition, and carries all the normal risks of editing postinst. [Discussion] Our timer on GCP runs every 5 minutes. This is necessary to support timely upgrades of gcp instances from standard ubuntu to ubuntu pro. We need to poll the metadata endpoint frequently to catch the license change in a timely manner. We exit as early as possible if there is nothing to be done for any given timer trigger. From our testing, this has minimal overall system performance impact. <TODO @chad.smith insert details and link to spreadsheet> [Changelog] * d/tools.postinst: - consider cloud to be "none" on any cloud-id error - purge old ua-messaging.timer/service files * systemd: - remove ua-messaging.timer/service - add new ua-timer.timer that runs every 2 hours - add new ua-license_check.timer that runs every 5 minutes only if activated by ua-license-check.path * New upstream release 27.3 - ros: + add beta support to enable ros and ros-updates + add support for "required services" so that esm-infra and esm-apps get auto-enabled when enabling ros or ros-updates + add support for "dependent services" so that user gets prompted to disable ros/ros-updates if they disable esm-infra/esm-apps - fips: + allow fips on GCP bionic now that optimized kernel is ready + disallow enabling fips on focal on clouds until cloud-optimized focal fips-certified kernel is ready (LP: #1939449, LP: #1939932) + print warning about generic fips kernel if cloud-id fails - cloud: + rely only on cloud-id to determine cloud type (LP: #1940131) + catch errors when determining cloud type (LP: #1938207) (GH: #1541) - azure: + bump IMDS API version to support Azure published images - cli: + collect-logs command that creates a tar file with debug-relevant logs and status info (GH: #463) + clean locks on exceptions more thoroughly to avoid false "Operation in progress" status messages + retain past service state after detach + shows better error message when a port value in a proxy is invalid - non-unicode locale support: + remove unicode-only characters from help file + don't print unicode-only characters in ua fix if non-utf8 locale (GH: #1463) - ua-timer.timer: + introduce a single systemd timer to handle ua recurring jobs + timer runs every 2 hours to support most frequent timer job + recurring job intervals are configurable in uaclient.conf + individual jobs are disabled if their interval is set to 0 - status job: + update ua status every 12 hours - messaging job: + update APT/MOTD ESM messaging every 6 hours - metering job: + disabled until infrastructure is ready + for attached machines only, periodically update contract server with status information for proper contract metering - ua-license-check.timer: + only runs on LTS GCP instances that are not attached + runs every 5 minutes to check if gcp instance has license required to auto-attach - logs: + fixes duplicate logging (GH: #553) - tests and support: + remove groovy integration tests + various improvements to integration tests
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1942929 Title: [SRU] ubuntu-advantage-tools (27.2.2 -> 27.3) Xenial, Bionic, Focal, Hirsute To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/1942929/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
