** Tags added: kernel-daily-bug
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016398
Title:
stacked overlay file system mounts that have chroot() called against
them appear to be getting locked (by the kernel most likely?)
Status in linux package in Ubuntu:
Confirmed
Status in linux source package in Focal:
Fix Committed
Status in linux source package in Jammy:
In Progress
Status in linux source package in Kinetic:
Won't Fix
Status in linux source package in Lunar:
Won't Fix
Status in linux source package in Mantic:
Won't Fix
Bug description:
[Impact]
Opened files reported in /proc/pid/map_files can be shows with the
wrong mount point using overlayfs with filesystem namspaces.
This incorrect behavior is fixed:
UBUNTU: SAUCE: overlayfs: fix incorrect mnt_id of files opened from
map_files
However, the fix introduced a new regression, the reference to the
original file stored in vma->vm_prfile is not properly released when
vma->vm_prfile is replaced with a new file.
This can cause a reference counter unbalance, leading errors such as
"target is busy" when trying to unmount overlayfs, even if the
filesystem has not active reference.
[Test case]
Reproducer provided by original bug reporter:
https://launchpadlibrarian.net/663151659/overlayfsscript_example
[Fix]
Fix by properly releasing the original file stored in vm_prfile.
[Regression potential]
This fix seems to solve the reported bug (verified with the
reproducer) and it doesn't seem to introduce other regressions or
behavior change.
However, we may experience regressions in overlayfs or potentially
other "target is busy" errors when unmounting overlayfs filesystems
with this fix applied, if there are still other corner cases not
covered properly.
[Original bug report]
Hi
BACKGROUND:
-----------
I have a set of scripts that debootstraps and builds vanilla Debian installs,
where the only modifications are what packages are installed. Then that becomes
the lower directory of an overlayfs mount, a tmpfs becomes the upperdir, and
then that becomes the chroot where config changes are made with more scripts.
This overlayfs is mounted in its own mount namespace as the script is unshare'd
Within these scripts, I make packages with a fork of checkinstall I
made which uses the / as the lowerdir, as overlayfs allows that, a
tmpfs as the upperdir, the install command is run with chroot, and
then the contents are copied.
THE ISSUE:
----------
I noticed that it appears that my ramdisks are remaining in memory, I have
isolated it out to the overlayfs mounts with the overlayfs as the lowerdir
where chroot is being called on it. I tried entering the namespace myself and
umounting it, and notice I get the error "Target is busy"
With the "Target Is Busy" issue, I tried to `lsof` and `fuser` and
`lsfd` as much as I could, but every user mode tool shows absolutely
NO files open at all
I was using Kubuntu 18.04, it was an old install and 32 bit, so I had
no upgrade path to be using more recent versions. Now I am on 23.04,
but I also think this is happening on a 22.10 laptop
DIAGNOSIS:
----------
I am able to isolate it out, and get this to replicate in the smallest script
that I could that is now attached. the chroot is important. I am able to
unmount the second overlayfs if chroot is never called. Nothing ever has to run
IN the chroot, trying to call a binary that does not exist, and then trying to
unmount it results in "target is busy" with no open files reportable by user
mode tools
FURTHER TESTING:
----------------
Trying to find out the version of Linux this was introduced, I made a script
that builds a very minimal ext4 file for a VM with a small debootstrapped
system, builds a kernel, and runs the kernel with QEMU..
but when I started, after all that, I can't replicate it at all with a
vanilla 6.2.0 kernel. I used the same config out of my /boot to the .config,
only tweaking a few options like SYSTEM_TRUSTED_KEYS. Trying to unmount the
second filesystem actually results in success
I then built a Kubuntu Lunar VM, and was able to replicate it on not
just my system. When I built and install the vanilla kernel with
bindeb-pkg, and then I boot the vanilla kernel, I can't replicate it,
the second filesystem unmounts. When I reboot the VM, and start the
default distribution kernel, I am able to replicate the issue, the
second filesystem being unable to unmount, with "target is busy" with
no files open according to standard usermode tools, when chroot is
called on it
I tried to look at patches in
git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/lunar
and git log --oneline --grep="sauce" but none of them really look relevant. I
could be VERY wrong in this matter though
SUMMARY:
--------
Long story short, (I am VERY sorry about the length:)
I make an overlayfs,
I make a second overlayfs with the first overlayfs as the second one's
lowerdir,
I chroot into the second overlayfs, and then later I can't unmount the second
one as it says "Target is busy", but the user mode tools don't show any files
open at all. If I mount the second overlayfs, and then unmount it without ever
chroot-ing into it I AM able to unmount it
The attached script should replicate the issue
This doesn't seem to happen on the vanilla Linux kernel, with the same
as close as possible config that Ubuntu uses
Thanks
ProblemType: Bug
DistroRelease: Ubuntu 23.04
Package: linux-image-6.2.0-20-generic 6.2.0-20.20
ProcVersionSignature: Ubuntu 6.2.0-20.20-generic 6.2.6
Uname: Linux 6.2.0-20-generic x86_64
ApportVersion: 2.26.1-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/controlC1: nerdopolis 3007 F.... wireplumber
/dev/snd/controlC0: nerdopolis 3007 F.... wireplumber
/dev/snd/seq: nerdopolis 3003 F.... pipewire
CasperMD5CheckResult: pass
CurrentDesktop: KDE
Date: Sat Apr 15 19:08:48 2023
InstallationDate: Installed on 2023-04-09 (6 days ago)
InstallationMedia: Kubuntu 23.04 "Lunar Lobster" - Daily amd64 (20230408)
MachineType: ASRock B650M PG Riptide
ProcFB: 0 amdgpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.2.0-20-generic
root=/dev/mapper/vgkubuntu-root ro quiet splash amdgpu.sg_display=0
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No
PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
linux-restricted-modules-6.2.0-20-generic N/A
linux-backports-modules-6.2.0-20-generic N/A
linux-firmware 20230323.gitbcdcfbcf-0ubuntu1
RfKill:
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/22/2023
dmi.bios.release: 5.26
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: 1.18
dmi.board.asset.tag: Default string
dmi.board.name: B650M PG Riptide
dmi.board.vendor: ASRock
dmi.board.version: Default string
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias:
dmi:bvnAmericanMegatrendsInternational,LLC.:bvr1.18:bd02/22/2023:br5.26:svnASRock:pnB650MPGRiptide:pvrDefaultstring:rvnASRock:rnB650MPGRiptide:rvrDefaultstring:cvnDefaultstring:ct3:cvrDefaultstring:skuDefaultstring:
dmi.product.family: Default string
dmi.product.name: B650M PG Riptide
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: ASRock
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2016398/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
