Your message dated Sun, 14 Sep 2025 23:18:25 +0200
with message-id <[email protected]>
and subject line Re: Bug#1115262: Error ... creating read-write layer...
permission denied
has caused the Debian Bug report #1115262,
regarding Error ... creating read-write layer... permission denied
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
1115262: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1115262
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: podman
Version: 4.3.1+ds1-8+deb12u1+b1
The tag2upload service, which uses podman via autopkgtest-virt-podman,
failed earlier today. I don't understand the cause, but I think it
must be some kind of race.
The symptoms are as follows.
The current failure looks like this. This happens every time we try
to "open the testbed" with autopkgtest-virt-podman:
Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]:
autopkgtest-virt-podman [17:47:48]: disabling init based on image label
Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]: <VirtSubproc>:
failure: ['podman', 'run', '--detach=true', '--volume',
'/tmp/autopkgtest-virt-docker.shared
.gmd824eq:/tmp/autopkgtest-virt-docker.shared.gmd824eq', '--network=host',
'localhost/autopkgtest/debian:bookworm', 'sleep', 'infinity'] failed (exit
status 125, stde
rr 'Error: creating container storage: creating read-write layer with ID
"dbec61978e3ec997def1e4ae0e8a7a94729ccff325aba6dbefce69bb396614e8": open
/srv/builder.tag2upl
oad.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/builder:
permission denied\n')
Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280275]: [t2u-oracled
tag2upload-builder-01.debian.org,1280275][2025-09-14T17:47:48] virt-server:
failed with error
exit status 12
We're using rootless podman containers.
I investigated by logging in as the service user:
tag2upload-builder@tag2upload-builder-01:~$ ls -al
/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/
total 12
drwxr-xr-x 3 tag2upload-builder tag2upload-builder 4096 Sep 11 03:55 .
dr-xr-sr-x 17 tag2upload-builder tag2upload-builder 4096 Sep 11 03:57 ..
drwx------ 4 100999 100999 4096 Sep 11 03:55 builder
tag2upload-builder@tag2upload-builder-01:~$
Immediately predecing the first failure, the logs contain this:
Sep 14 15:30:27 tag2upload-oracle-01 using-these[1261274]: Connection to
tag2upload-builder-01.debian.org closed by remote host.
Sep 14 17:47:25 tag2upload-oracle-01 using-these[841]: [t2u-oracled
tag2upload-builder-01.debian.org,841][2025-09-14T17:47:25] group_leader
worker=716680: died due to
fatal signal PIPE
This was probably due to a transient network problem between two of
the tag2upload service's hosts. The process reported there as
receiving SIGPIPE was the parent of autopkgtest-virt-podman. I don't
think autopkgtest-virt-podman would have got any signal as a result of
this event. It would have seen an EOF on its stdin and if it
attempted to write to its stdout it would probably have seen
EPIPE/SIGPIPE.
I conjecture that podman (or autopkgtest-virt-podman) leaves a broken,
stuck, state if things fail at the wrong moemnt.
I tried to run our image rebuild script in the hope of fixing things,
but without success:
+ podman image prune --force
Error: openfdat
/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir/d3c5473984fc03ec91dcf6a31e183f1ba34c5aebacbd73d6b973c94b4a14d869/home/builder:
permission denied
I tried moving the directory out with mv (transcript below) but that
didn't work.
I then did this:
tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs$
mv dir dir.broken
tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs$
I then ran rm -rf dir.broken to delete what I could, to try to recover
disk space. This seems to have allowed the system to work again.
I will file a DSA ticket to ask them to remvoe the `dir.broken`.
Ian.
2a565a034c8cc4c53501
9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken
tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$
rm -rf 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken
rm: cannot remove
'9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken/sys':
Permission denied
...
rm: cannot remove
'9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken/var/cache/man/ko/cat8':
Permission denied
...
tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$
mkdir /srv/builder.tag2upload.debian.org/broken
tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$
mv 9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken
/srv/builder.tag2upload.debian.org/broken/.
mv: cannot move
'9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken' to
'/srv/builder.tag2upload.debian.org/broken/./9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501.broken':
Permission denied
tag2upload-builder@tag2upload-builder-01:/srv/builder.tag2upload.debian.org/home/.local/share/containers/storage/vfs/dir$
ls
--
Ian Jackson <[email protected]> These opinions are my own.
Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.
--- End Message ---
--- Begin Message ---
Hi,
On 2025-09-14 20:46, Ian Jackson wrote:
> Package: podman
> Version: 4.3.1+ds1-8+deb12u1+b1
>
> The tag2upload service, which uses podman via autopkgtest-virt-podman,
> failed earlier today. I don't understand the cause, but I think it
> must be some kind of race.
>
> The symptoms are as follows.
>
> The current failure looks like this. This happens every time we try
> to "open the testbed" with autopkgtest-virt-podman:
>
> Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]:
> autopkgtest-virt-podman [17:47:48]: disabling init based on image label
> Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280276]: <VirtSubproc>:
> failure: ['podman', 'run', '--detach=true', '--volume',
> '/tmp/autopkgtest-virt-docker.shared
> .gmd824eq:/tmp/autopkgtest-virt-docker.shared.gmd824eq', '--network=host',
> 'localhost/autopkgtest/debian:bookworm', 'sleep', 'infinity'] failed (exit
> status 125, stde
> rr 'Error: creating container storage: creating read-write layer with ID
> "dbec61978e3ec997def1e4ae0e8a7a94729ccff325aba6dbefce69bb396614e8": open
> /srv/builder.tag2upl
> oad.debian.org/home/.local/share/containers/storage/vfs/dir/9d2fecf88515328d21eeda5a7a1e41e9d83daa26d1262a565a034c8cc4c53501/home/builder:
> permission denied\n')
> Sep 14 17:47:48 tag2upload-oracle-01 using-these[1280275]: [t2u-oracled
> tag2upload-builder-01.debian.org,1280275][2025-09-14T17:47:48] virt-server:
> failed with error
> exit status 12
The issue is actually my fault, not the fault of podman. I changed the
subuid/subgid of the tag2upload-builder user [1]. I stopped the service
before doing that change, but didn't realize that some files in the home
directory were using the old subuid/subgid.
I am therefore closing the bug.
Regards
Aurelien
[1]
https://salsa.debian.org/dsa-team/mirror/dsa-puppet/-/commit/dac624978b2722e05492d0bd5f8011f0355f4df6
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
[email protected] http://aurel32.net
--- End Message ---
_______________________________________________
Pkg-go-maintainers mailing list
[email protected]
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-go-maintainers
