Public bug reported:
Summary
A CephFS kernel client that the MDS had blocklisted was unmounted when its last
user exited.
The unmount timed out, and generic_shutdown_super warned "VFS: Busy inodes
after unmount of
ceph". About seven hours later, a plain read of /proc/locks hit a page fault in
lock_get_status. The reading task died holding the file-lock lists' lock. From
then on every
flock(), fcntl() lock and /proc/locks read on the machine blocked in D state,
interactive
logins hung on /var/log/wtmp, and a normal reboot hung. A hard power cycle was
needed.
System
- Kernel 7.0.0-38-generic #38-Ubuntu, x86_64; Supermicro AS -8125GS-TNHR (2x
AMD EPYC, 8x H100).
- Tainted O (NVIDIA modules).
- CephFS kernel client mounted at /data from a Ceph cluster we don't run
(Tentacle MDS).
Sequence (UTC)
1. Oct 16 08:08: the MDS evicted and blocklisted client 3689341. Our recovery
script unmounted
/data lazily and mounted it again as a new client. Processes that had files
open on the old
mount, among them a user's tmux server, kept the old superblock alive.
2. Oct 22 23:26:39: that tmux server exited and was the old mount's last user.
In do_exit, then
cleanup_mnt, deactivate_super and ceph_kill_sb, the kernel logged
`ceph: [3689341]: umount timed out, 0` and then
`WARNING: fs/super.c:654 at generic_shutdown_super ... VFS: Busy inodes
after unmount of ceph`.
3. Oct 23 06:45:41: `awk` read /proc/locks:
`BUG: unable to handle page fault for address: 0000006600667248`,
`RIP: lock_get_status+0x124/0x340`, called from locks_show, seq_read_iter and
proc_reg_read_iter. The fault address is R13 (0x6600667220) + 0x28.
Every file-lock operation hung after that.
Both traces are attached (kernel-trace.txt).
Cause
Our recovery runs `umount -f` on the dead CephFS mount. ceph_umount_begin()
shuts the mount
down even though the unmount itself then fails as busy. From then on,
ceph_lock() and
ceph_flock() return -ESTALE for every request, unlocks included, before they
drop the lock's
local record.
- On close, the VFS ignores that error, warns "Leaked FLOCK lock" (or POSIX)
and frees the file.
- The lock record stays on file_lock_list, with flc_file pointing at the freed
file, and keeps
a reference on its inode. That reference is why the final unmount reported
busy inodes.
- At lock_get_status+0x124, the kernel loads inode->i_sb, where inode =
flc->flc_file->f_inode was read from that freed, reused memory. The faulting
instruction was
matched against the 7.0.0-38 image.
The journal of that boot shows it happening. After each of three forced
remounts on this node
(Oct 14, 15 and 16), the kernel logged "Leaked FLOCK lock on dev=0x0:0x3c /
0x42 / 0x43", six
in all, 25 to 45 minutes after each remount. The code involved is the same in
upstream v7.0
and mainline. It dates from commit 5d6451b1489a ("ceph: shut down access to
inode when async
create fails", v5.16).
A fix for fs/ceph/locks.c is going to ceph-devel: drop the local record on
unlock before
returning -ESTALE, the way the CEPH_I_ERROR_FILELOCK path already does. We'll
link it here.
We have not reproduced this on purpose. A likely recipe: take an flock() lock
on a CephFS file,
`umount -f` the mount while the holder stays alive (it fails as busy but shuts
the mount
down), end the holder, then read /proc/locks. Expect "Leaked FLOCK lock" in
dmesg, a stale
/proc/locks entry, and a use-after-free on a KASAN kernel.
Expected
An unlock on a shut-down CephFS mount should still drop the local lock record,
so no record
outlives its file.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: ceph resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172041
Title:
ceph: after a blocklisted client's unmount leaves "Busy inodes",
reading /proc/locks oopses in lock_get_status
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2172041/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
