Public bug reported:
Impact]
On systems that utilize both uprobes and perf_events style profiling, it
is possible to hit a panic in the uprobe_free_utask code. This occurs
during process exit. If the profiler fires while uprobe_free_utask is
in the process of cleaning up the utask, the NMI may read freed memory
because the cleanup code frees the utask before setting its pointer to
NULL. This submitter has encountered the problem on systems running
workloads without intentionally trying to trigger the problem.
The stacks look something like this:
RIP: 0010:is_uprobe_at_func_entry+0x28/0x80
...
? die_addr+0x36/0x90
? exc_general_protection+0x217/0x420
? asm_exc_general_protection+0x26/0x30
? is_uprobe_at_func_entry+0x28/0x80
perf_callchain_user+0x20a/0x360
get_perf_callchain+0x147/0x1d0
bpf_get_stackid+0x60/0x90
bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b
? __smp_call_single_queue+0xad/0x120
bpf_overflow_handler+0x75/0x110
...
asm_sysvec_apic_timer_interrupt+0x1a/0x20
RIP: 0010:__kmem_cache_free+0x1cb/0x350
...
? uprobe_free_utask+0x62/0x80
? acct_collect+0x4c/0x220
uprobe_free_utask+0x62/0x80
mm_release+0x12/0xb0
do_exit+0x26b/0xaa0
__x64_sys_exit+0x1b/0x20
do_syscall_64+0x5a/0x80
The person who reported the issue upstream provided this reproducer.
(Run each command in a separate terminal):
# while :; do bpftrace -e 'uprobe:/bin/ls:_start { printf("hit\n"); }' -c
ls; done
# bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
However, since the binutils are stripped on some of the releases where I
tested this, I ran the following instead:
# while :; do bpftrace -e 'uprobe:libc:malloc { printf("hit\n"); }' -c ls;
done
# bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
[Backport]
The fix is upstream as commit b583ef82b671 ("uprobes: Fix race in
uprobe_free_utask")
However this patch was massaged by stable for its inclusion in 6.12,
6.6, and 6.1. Instead of re-doing stable's conflict resolution, take
the patch directly from 6.6.x instead, at commit eff00c5e29ab.
This patch is in stable as of 6.12.19, 6.6.83, and 6.1.131.
[Test]
I've run the provided reproducer and validated that I can reproduce the
problem without the patch applied and that I cannot reproduce it again
once I have applied the patch.
[Potential Regression]
The regression potential here seems quite low. The fix has been
upstream for a couple releases and no subsequent issues have been
reported. It makes no functional change beyond ensuring that the utask
pointer is set to NULL before the utask structure itself is freed. The
dereference and free occur on the same cpu.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: patch patch-accepted-upstream
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2104210
Title:
uprobe-related panics during profiling
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2104210/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
