Package: podman Version: 4.3.1+ds1-8+deb12u1+b1 The tag2upload service, which uses podman via autopkgtest-virt-podman, failed earlier today. I don't understand the cause, but I think it must be some kind of race.
The symptoms are as follows. The current failure looks like this. This happens every time we try to "open the testbed" with autopkgtest-virt-podman: Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]: autopkgtest-virt-podman [17:47:48]: disabling init based on image label Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]: <VirtSubproc>: failure: ['podman', 'run', '--detach=true', '--volume', '/tmp/autopkgtest-virt-docker.shared .gmd824eq:/tmp/autopkgtest-virt-docker.shared.gmd824eq', '--network=host', 'localhost/autopkgtest/debian:bookworm', 'sleep', 'infinity'] failed (exit status 125, stde rr 'Error: creating container storage: creating read-write layer with ID "dbec61978e3ec997def1e4ae0e8a7a94729ccff325aba6dbefce69bb396614e8": open /srv/builder.tag2upl oad.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/builder: permission denied\n') Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280275]: [t2u-oracled tag2upload-builder-01.debian.org,1280275][2025-09-14T17:47:48] virt-server: failed with error exit status 12 We're using rootless podman containers. I investigated by logging in as the service user: tag2upload-builder@tag2upload-builder-01:~$ ls -al /srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/ total 12 drwxr-xr-x 3 tag2upload-builder tag2upload-builder 4096 Sep 11 03:55 . dr-xr-sr-x 17 tag2upload-builder tag2upload-builder 4096 Sep 11 03:57 .. drwx------ 4 100999 100999 4096 Sep 11 03:55 builder tag2upload-builder@tag2upload-builder-01:~$ Immediately predecing the first failure, the logs contain this: Sep 14 15:30:27 tag2upload-oracle-01 using-these[1261274]: Connection to tag2upload-builder-01.debian.org closed by remote host. Sep 14 17:47:25 tag2upload-oracle-01 using-these[841]: [t2u-oracled tag2upload-builder-01.debian.org,841][2025-09-14T17:47:25] group_leader worker=716680: died due to fatal signal PIPE This was probably due to a transient network problem between two of the tag2upload service's hosts. The process reported there as receiving SIGPIPE was the parent of autopkgtest-virt-podman. I don't think autopkgtest-virt-podman would have got any signal as a result of this event. It would have seen an EOF on its stdin and if it attempted to write to its stdout it would probably have seen EPIPE/SIGPIPE. I conjecture that podman (or autopkgtest-virt-podman) leaves a broken, stuck, state if things fail at the wrong moemnt. I tried to run our image rebuild script in the hope of fixing things, but without success: + podman image prune --force Error: openfdat /srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir/d3c5473984fc03ec91dcf6a31e183f1ba34c5aebacbd73d6b973c94b4a14d869/home/builder: permission denied I tried moving the directory out with mv (transcript below) but that didn't work. I then did this: tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs$ mv dir dir.broken tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs$ I then ran rm -rf dir.broken to delete what I could, to try to recover disk space. This seems to have allowed the system to work again. I will file a DSA ticket to ask them to remvoe the `dir.broken`. Ian. 2a565a034c8cc4c53501 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ rm -rf 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken rm: cannot remove '9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken/sys': Permission denied ... rm: cannot remove '9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken/var/cache/man/ko/cat8': Permission denied ... tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ mkdir /srv/builder.tag2upload.debian.org/broken tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ mv 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken /srv/builder.tag2upload.debian.org/broken/. mv: cannot move '9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken' to '/srv/builder.tag2upload.debian.org/broken/./9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken': Permission denied tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$ ls -- Ian Jackson <[email protected]> These opinions are my own. Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.
