On Wed, Mar 11, 2026 at 8:53 PM David Laight <[email protected]> wrote: > > On Wed, 11 Mar 2026 12:54:26 +0100 > Peter Zijlstra <[email protected]> wrote: > > > On Wed, Mar 11, 2026 at 07:52:47PM +0800, Yafang Shao wrote: > > > Recently, we resolved a latency spike issue caused by concurrently running > > > bpftrace processes. The root cause was high contention on the ftrace_lock > > > due to optimistic spinning. We can optimize this by disabling optimistic > > > spinning for ftrace_lock. > > > > > > While semaphores may present similar challenges, I'm not currently aware > > > of > > > specific instances that exhibit this exact issue. Should we encounter > > > problematic semaphores in production workloads, we can address them at > > > that > > > time. > > > > > > PATCH #1: introduce slow_mutex_[un]lock to disable optimistic spinning > > > PATCH #2: add variant for rtmutex > > > PATCH #3: disable optimistic spinning for ftrace_lock > > > > > > > So I really utterly hate this. > > Yep... > Adding the extra parameter is likely to have a measurable impact > on everything else. > > The problematic path is obvious: find_kallsyms_symbol+142 > module_address_lookup+104 > kallsyms_lookup_buildid+203 > kallsyms_lookup+20 > print_rec+64 > t_show+67 > seq_read_iter+709 > seq_read+165 > vfs_read+165 > ksys_read+103 > __x64_sys_read+25 > do_syscall_64+56 > entry_SYSCALL_64_after_hwframe+100 > > The code needs to drop the ftrace_lock across t_show.
It's unclear whether we can safely release ftrace_lock within t_show — doing so would probably necessitate a major redesign of the current implementation. > > Although there is a bigger issue of why on earth the code is reading the > list of filter functions at all - never mind all the time. bpftrace reads the complete list of available functions into userspace, then performs matching against the target function to determine if it is traceable. > I'll do it by hand when debugging, but I'd have though anything using bpf > will know exactly where to add its hooks. -- Regards Yafang
