** Description changed: [ Impact ] This has been affecting minimal and base pipelines for cloud images for both oracular and noble and blocking publication for these images. The oracular and noble images fail lxd related tests because lxd installation cannot complete with the following error `ConnectionResetError: [Errno 104] Connection reset by peer`. This error is intermittent. The proposed upload allows retries of the installation to mitigate availability issues from the snapstore. [ Test Plan ] To reproduce the bug you can launch a container from either oracular or noble like so: `lxc init ubuntu-<SUITE>-daily:<SUITE> test && lxc start - test && lxc exec test bash` where SUITE=oracular or noble. Since this - error is intermittent, this reproducer is also the same. + test && lxc exec test bash` where SUITE=oracular or noble. + + Because the error is intermittent and happens due to unavailability of + the snapcraft API, you can simulate this by redirecting requests to + api.snapcraft.io. Add the following to the /etc/hosts: `127.0.0.1 + api.snapcraft.io` and this make the snap install is not reaching the + snapcraft API. Upon connecting to the container you can run `lxd init --auto --storage- backend dir` which results into the error `ConnectionResetError: [Errno 104] Connection reset by peer` intermittently. From journalctl lxd logs, the following appears: `error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh":` The package with the proposed changes will allow the retry loop to attempt installing lxd in the oracular/noble containers in cases where connection to api.snapcraft.io is not achieved in the first attempt. [ Where problems could occur ] The possible effect of this change is posting to snapcraft more frequently which could increase the traffic to `https://api.snapcraft.io`. The number of retries is however limited and the installation failure intermittent so the additional retries should not have a significant impact on Snapcraft. The package itself does not have any changes as the retry targets only the installer. [ Other Info ] The same change in lxd-installer has been accepted for plucky and is has been in build pipelines for our plucky images for a few weeks now; these images do not fail on the lxd-installer anymore. [Original Description] Description: Ubuntu Plucky Puffin (development branch) (*LXD container) Release: 25.04 Image serial: 20250217 Source package: https://launchpad.net/ubuntu/plucky/+package/lxd-installer Package version: ``` $ apt-cache policy lxd-installer lxd-installer: Installed: 10 Candidate: 10 Version table: *** 10 100 100 /var/lib/dpkg/status ``` Expected behaviour: `lxd init --auto --storage-backend dir` command would trigger then `lxd-installer` shim and`lxd` would be installed in the container. What happened instead: ``` 10:19:41 2025-02-17 17:19:41,454 [INFO] test_framework.tests.lxd_start_stop: initializing lxd 10:19:41 Installing LXD snap, please be patient. 10:19:41 Traceback (most recent call last): 10:19:41 File "<string>", line 1, in <module> 10:19:41 import socket; s=socket.socket(socket.AF_UNIX); s.connect("/run/lxd-installer.socket"); s.send(b"x"); s.recv(1) 10:21:12 ~~~~~~^^^ 10:21:12 ConnectionResetError: [Errno 104] Connection reset by peer ``` The command that triggers the shim is `lxd init --auto --storage-backend dir` Reproducer: This failure is intermittent, and I have had a hard time getting a reliable reproducer for it. The best that I have got is: ``` lxc init ubuntu-minimal-daily:plucky test #20250217 serial was pulled lxc start test lxc exec test bash <inside the shell> lxd init --auto --storage-backend dir # confirm with 'y' ``` The error should appear almost instantly. If it does not, stop & delete the instance and repeat the above. *LXD team suggested reproducer: ``` $ while lxc launch ubuntu-minimal-daily:plucky test && lxc exec --force-noninteractive test -- lxc --version && lxc delete -f test; do sleep 1; done ``` Logs: `journalctl --grep lxd`: ``` root@genuine-satyr:~# journalctl --grep lxd Feb 25 23:35:52 genuine-satyr useradd[324]: add 'ubuntu' to group 'lxd' Feb 25 23:35:52 genuine-satyr useradd[324]: add 'ubuntu' to shadow group 'lxd' Feb 25 23:35:53 genuine-satyr systemd[1]: Starting lxd-installer.socket - Helper to install lxd snap on demand... Feb 25 23:35:53 genuine-satyr systemd[1]: Listening on lxd-installer.socket - Helper to install lxd snap on demand. Feb 25 23:35:56 genuine-satyr cloud-init[262]: Cloud-init v. 25.1~3geb1965a4-0ubuntu1 finished at Tue, 25 Feb 2025 23:35:56 +0000. Datasource DataSourceLXD. Up 5.51 seconds Feb 25 23:36:00 genuine-satyr systemd[1]: Created slice system-lxd\x2dinstaller.slice - Slice /system/lxd-installer. Feb 25 23:36:00 genuine-satyr systemd[1]: Started lxd-installer@0-503-0.service - Helper to install lxd snap on demand (PID 503/UID 0). Feb 25 23:36:00 genuine-satyr lxd-installer-service[504]: + [ lxd-installer-service = lxd-installer-service ] Feb 25 23:36:00 genuine-satyr lxd-installer-service[511]: + lxd_channel Feb 25 23:36:00 genuine-satyr lxd-installer-service[504]: + snap install lxd --channel=5.21/stable/ubuntu-25.04 Feb 25 23:36:00 genuine-satyr snapd[348]: api_snaps.go:467: Installing snap "lxd" revision unset Feb 25 23:36:00 genuine-satyr lxd-installer-service[512]: error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh": Feb 25 23:36:00 genuine-satyr systemd[1]: lxd-installer@0-503-0.service: Main process exited, code=exited, status=1/FAILURE Feb 25 23:36:00 genuine-satyr systemd[1]: lxd-installer@0-503-0.service: Failed with result 'exit-code'. ``` `journalctl` around failure: ``` Feb 25 23:40:12 test systemd[1]: Created slice system-lxd\x2dinstaller.slice - Slice /system/lxd-installer. Feb 25 23:40:12 test systemd[1]: Started lxd-installer@0-508-0.service - Helper to install lxd snap on demand (PID 508/UID 0). Feb 25 23:40:12 test lxd-installer-service[509]: + [ lxd-installer-service = lxd-installer-service ] Feb 25 23:40:12 test lxd-installer-service[509]: + snap wait system seed.loaded Feb 25 23:40:12 test lxd-installer-service[516]: + lxd_channel Feb 25 23:40:12 test lxd-installer-service[516]: + track= Feb 25 23:40:12 test lxd-installer-service[516]: + [ -r /etc/os-release ] Feb 25 23:40:12 test lxd-installer-service[516]: + . /etc/os-release Feb 25 23:40:12 test lxd-installer-service[516]: + PRETTY_NAME=Ubuntu Plucky Puffin (development branch) Feb 25 23:40:12 test lxd-installer-service[516]: + NAME=Ubuntu Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION_ID=25.04 Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION=25.04 (Plucky Puffin) Feb 25 23:40:12 test lxd-installer-service[516]: + VERSION_CODENAME=plucky Feb 25 23:40:12 test lxd-installer-service[516]: + ID=ubuntu Feb 25 23:40:12 test lxd-installer-service[516]: + ID_LIKE=debian Feb 25 23:40:12 test lxd-installer-service[516]: + HOME_URL=https://www.ubuntu.com/ Feb 25 23:40:12 test lxd-installer-service[516]: + SUPPORT_URL=https://help.ubuntu.com/ Feb 25 23:40:12 test lxd-installer-service[516]: + BUG_REPORT_URL=https://bugs.launchpad.net/ubuntu/ Feb 25 23:40:12 test lxd-installer-service[516]: + PRIVACY_POLICY_URL=https://www.ubuntu.com/legal/terms-and-policies/privacy-policy Feb 25 23:40:12 test lxd-installer-service[516]: + UBUNTU_CODENAME=plucky Feb 25 23:40:12 test lxd-installer-service[516]: + LOGO=ubuntu-logo Feb 25 23:40:12 test lxd-installer-service[516]: + track=5.21 Feb 25 23:40:12 test lxd-installer-service[516]: + [ -n 5.21 ] Feb 25 23:40:12 test lxd-installer-service[516]: + [ -n 25.04 ] Feb 25 23:40:12 test lxd-installer-service[516]: + echo 5.21/stable/ubuntu-25.04 Feb 25 23:40:12 test lxd-installer-service[509]: + CHANNEL=5.21/stable/ubuntu-25.04 Feb 25 23:40:12 test lxd-installer-service[509]: + [ -z 5.21/stable/ubuntu-25.04 ] Feb 25 23:40:12 test lxd-installer-service[509]: + snap install lxd --channel=5.21/stable/ubuntu-25.04 Feb 25 23:40:12 test snapd[345]: api_snaps.go:467: Installing snap "lxd" revision unset Feb 25 23:40:12 test snapd[345]: store_download.go:142: no host system xdelta3 available to use deltas Feb 25 23:40:12 test lxd-installer-service[517]: error: cannot install "lxd": Post "https://api.snapcraft.io/v2/snaps/refresh": Feb 25 23:40:12 test lxd-installer-service[517]: context canceled Feb 25 23:40:12 test systemd[1]: lxd-installer@0-508-0.service: Main process exited, code=exited, status=1/FAILURE Feb 25 23:40:12 test systemd[1]: lxd-installer@0-508-0.service: Failed with result 'exit-code'. Feb 25 23:40:14 test snapd[345]: overlord.go:518: Released state lock file Feb 25 23:40:14 test snapd[345]: daemon stop requested to wait for socket activation Feb 25 23:40:14 test systemd[1]: snapd.service: Deactivated successfully. Feb 25 23:40:14 test systemd[1]: snapd.service: Consumed 1.014s CPU time, 37.6M memory peak. ```
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2100564 Title: lxd-installer shim fails to install with snapstore error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxd-installer/+bug/2100564/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
