Introduction.
The affected system is stable/14, amd64.
The kernel is custom, it's configured with INVARIANTS.
The problem started to happen rather reliably after a recent upgrade of
packages. I suspect that the trigger is in linux-nvidia-libs-570.124.04, but
the bug is in FreeBSD Linux emulation.
The reason for my suspicion is that the crash happens when starting a graphical
Linux application in a Linux jail. And the crash involves a graphics-related
character device.
Just in case, the jail itself, including the application, hasn't been changed.
Also, I haven't touched the base system recently.
Details.
VNASSERT failed: old > 0 not true at sys/kern/vfs_subr.c:3361 (vrefact)
0xfffff802945df380: type VCHR state VSTATE_CONSTRUCTED op 0xffffffff8127b648
usecount 1, writecount 0, refcount 39 seqc users 0 rdev 0xfffff8004565f400
hold count flags ()
flags ()
lock type devfs: UNLOCKED
dev drm/128
panic: vrefact: wrong use count 0
cpuid = 1
time = 1742796535
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8061eadb = db_trace_self_wrapper+0x2b/frame
0xfffffe02476a0780
kdb_backtrace() at 0xffffffff80956a57 = kdb_backtrace+0x37/frame
0xfffffe02476a0830
vpanic() at 0xffffffff80907629 = vpanic+0x169/frame 0xfffffe02476a0970
panic() at 0xffffffff80907403 = panic+0x43/frame 0xfffffe02476a09d0
vrefact() at 0xffffffff809f08e4 = vrefact+0xb4/frame 0xfffffe02476a09f0
fgetvp_lookup() at 0xffffffff808ac718 = fgetvp_lookup+0x88/frame
0xfffffe02476a0a30
namei_setup() at 0xffffffff809e07ba = namei_setup+0x15a/frame 0xfffffe02476a0a80
namei_emptypath() at 0xffffffff809e0499 = namei_emptypath+0x49/frame
0xfffffe02476a0ae0
namei() at 0xffffffff809e029f = namei+0x66f/frame 0xfffffe02476a0b40
linux_kern_statat() at 0xffffffff8a09d24c = linux_kern_statat+0xfc/frame
0xfffffe02476a0c70
linux_newfstatat() at 0xffffffff8a09cfed = linux_newfstatat+0x6d/frame
0xfffffe02476a0e00
amd64_syscall() at 0xffffffff80c79f79 = amd64_syscall+0x189/frame
0xfffffe02476a0f30
fast_syscall_common() at 0xffffffff80c4fb9b = fast_syscall_common+0xf8/frame
0xfffffe02476a0f30
--- syscall (262, Linux ELF64, linux_newfstatat), rip = 0x813f13eee, rsp =
0x7fffffffbd28, rbp = 0 ---
As far as I understand, there is a Linux fstatat system call with AT_EMPTY_PATH
flag and the file descriptor of opened /dev/drm/128 device.
Given that the crash happens in fgetvp_lookup -> vrefact, I think that it's
unlikely that there is a problem in that call path.
I believe that the problem is elsewhere in the Linux emulation code for working
with character devices.
I think that the panic means that the corresponding file descriptor was open but
the associated vnode had usecount of zero.
It looks like DTYPE_DEV (11) is used only in the linuxkpi code, e.g.,
linux_dev_fdopen.
Some info from kgdb.
(kgdb) bt
#0 __curthread () at sys/amd64/include/pcpu_aux.h:57
#1 doadump (textdump=textdump@entry=1) at sys/kern/kern_shutdown.c:423
#2 0xffffffff80907121 in kern_reboot (howto=260) at
sys/kern/kern_shutdown.c:541
#3 0xffffffff80907698 in vpanic (fmt=0xffffffff80e35cf8 "%s: wrong use count
%d", ap=0xfffffe01adc909b0) at sys/kern/kern_shutdown.c:1021
#4 0xffffffff80907403 in panic (fmt=<unavailable>) at
sys/kern/kern_shutdown.c:945
#5 0xffffffff809f08e4 in vrefact (vp=0xfffff8035b4bb700) at
sys/kern/vfs_subr.c:3361
#6 0xffffffff808ac718 in fgetvp_lookup (ndp=ndp@entry=0xfffffe01adc90b58,
vpp=vpp@entry=0xfffffe01adc90ac8) at sys/kern/kern_descrip.c:3134
#7 0xffffffff809e07ba in namei_setup (ndp=ndp@entry=0xfffffe01adc90b58,
dpp=dpp@entry=0xfffffe01adc90ac8, pwdp=pwdp@entry=0xfffffe01adc90ac0) at
sys/kern/vfs_lookup.c:383
#8 0xffffffff809e0499 in namei_emptypath (ndp=ndp@entry=0xfffffe01adc90b58) at
sys/kern/vfs_lookup.c:466
#9 0xffffffff809e029f in namei (ndp=ndp@entry=0xfffffe01adc90b58) at
sys/kern/vfs_lookup.c:687
#10 0xffffffff8a09d24c in linux_kern_statat (td=0xfffff804d50d7000, flag=16384,
fd=9, path=0x813fd846f <error: Cannot access memory at address 0x813fd846f>,
pathseg=UIO_USERSPACE, sbp=sbp@entry=0xfffffe01adc90c80)
at sys/compat/linux/linux_stats.c:103
#11 0xffffffff8a09cfed in linux_newfstatat (td=<unavailable>, td@entry=<error
reading variable: value is not available>, args=0xfffff804d50d7400,
args@entry=<error reading variable: value is not available>)
at sys/compat/linux/linux_stats.c:620
#12 0xffffffff80c79f79 in syscallenter (td=0xfffff804d50d7000) at
sys/amd64/amd64/../../kern/subr_syscall.c:191
#13 amd64_syscall (td=0xfffff804d50d7000, traced=<optimized out>) at
sys/amd64/amd64/trap.c:1206
(kgdb) p *vp
$1 = {v_type = VCHR, v_state = VSTATE_CONSTRUCTED, v_irflag = 0, v_seqc = 0,
v_nchash = 1973399077, v_hash = 56314807, v_op = 0xffffffff8127b648
<devfs_specops>, v_data = 0xfffff80055005200, v_mount = 0xfffffe0150b46100,
v_nmntvnodes = {
tqe_next = 0xfffff8038000da80, tqe_prev = 0xfffff8035b4bb8e8},
{v_mountedhere = 0xfffff800452b9400, v_unpcb = 0xfffff800452b9400, v_rdev =
0xfffff800452b9400, v_fifoinfo = 0xfffff800452b9400}, v_hashlist = {le_next =
0x0, le_prev = 0x0},
v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last =
0xfffff8035b4bb758}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name =
0xffffffff80d1cf3c "devfs", lo_flags = 116588544, lo_data = 0, lo_witness = 0x0},
lk_lock = 1, lk_exslpfail = 0, lk_pri = 64, lk_timo = 51}, v_interlock =
{lock_object = {lo_name = 0xffffffff80db24c1 "vnode interlock", lo_flags =
16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, v_vnlock =
0xfffff8035b4bb770,
v_vnodelist = {tqe_next = 0xfffff8035b4bbc40, tqe_prev = 0xfffff80369f48280},
v_lazylist = {tqe_next = 0x0, tqe_prev = 0x0}, v_bufobj = {bo_lock =
{lock_object = {lo_name = 0xffffffff80df4394 "bufobj interlock", lo_flags =
86179840,
lo_data = 0, lo_witness = 0x0}, rw_lock = 1}, bo_ops =
0xffffffff812b7190 <buf_ops_bio>, bo_object = 0x0, bo_synclist = {le_next = 0x0,
le_prev = 0x0}, bo_private = 0xfffff8035b4bb700, bo_clean = {bv_hd = {tqh_first
= 0x0,
tqh_last = 0xfffff8035b4bb828}, bv_root = {pt_root = 0x1}, bv_cnt = 0},
bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8035b4bb848}, bv_root =
{pt_root = 0x1}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_domain = 0,
bo_bsize = 512}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl =
{rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff8035b4bb890}, rl_currdep =
0x0}, v_holdcnt = 32, v_usecount = 1, v_iflag = 0, v_vflag = 0, v_mflag = 0,
v_dbatchcpu = -1, v_writecount = 0, v_seqc_users = 0}
(kgdb) p *fp
$3 = {f_flag = 3, f_count = 3, f_data = 0xfffff807120b5480, f_ops =
0xffffffff84b46390 <linuxfileops>, f_vnode = 0xfffff8035b4bb700, f_cred =
0xfffff8036f967d00, f_type = 11, f_vnread_flags = 0, {f_seqcount = {0, 0},
f_pipegen = 0},
f_nextoff = {0, 0}, f_vnun = {fvn_cdevpriv = 0x0, fvn_advice = 0x0}, f_offset
= 0}
(kgdb) p *ndp
$5 = {ni_dirp = 0x813fd846f <error: Cannot access memory at address
0x813fd846f>, ni_segflg = UIO_USERSPACE, ni_rightsneeded = 0xffffffff812005f0
<cap_fstat_rights>, ni_startdir = 0x0, ni_rootdir = 0xfffff8003a922c40,
ni_topdir = 0xfffff8003a922c40, ni_dirfd = 9, ni_lcf = 0, ni_filecaps =
{fc_rights = {cr_rights = {144123984168878079, 288230376153808895}}, fc_ioctls =
0x0, fc_nioctls = -1, fc_fcntls = 120}, ni_vp = 0x0, ni_dvp = 0xffffffffffffffff,
ni_resflags = 4, ni_debugflags = 3, ni_loopcnt = 0, ni_pathlen = 1, ni_next =
0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>,
ni_cnd = {cn_flags = 262596, cn_cred = 0xfffff8068c56b200, cn_nameiop = LOOKUP,
cn_lkflags = -1, cn_pnbuf = 0xfffff8002b11ec00 "", cn_nameptr =
0xfffff8002b11ec00 "", cn_namelen = -1}, ni_cap_tracker = {tqh_first = 0x0,
tqh_last = 0xfffffe01adc90c08}, ni_dvp_seqc = 2915634432, ni_vp_seqc = 4294966785}
I tried to look at linux_dev_fdopen() and other code in
sys/compat/linuxkpi/common/src/linux_compat.c, but couldn't make much progress yet.
I have the crash dump, so if there is anything else I can provide or look at...
Thank you.
--
Andriy Gapon
