On Wed, 8 Jul 2026 21:37:53 +0900 Jeongho Choi <[email protected]> wrote:
> Hello, > > We are seeing a reproducible kernel panic related to the tracing code > when it fails to read user-space memory. > > The issue was originally reported through the Android/Google Issue > Tracker, and we were advised to report it to the upstream trace mailing > list because the affected code is upstream. > > Environment: > > Architecture: arm64 > Kernel: Linux 6.18.21 > Base: Android Common Kernel (android17-6.18) > Affected area: kernel/trace/ > > The relevant error/panic log is: > [48916.569148] [9: lmkd: 536] Error: Too many tries to read > user space > [48916.569156] [9: lmkd: 536] WARNING: CPU: 9 PID: 536 at > kernel/trace/trace.c:7374 trace_user_fault_read+0x334/0x360 > [48916.569443] [9: lmkd: 536] CPU: 9 UID: 1069 PID: 536 Comm: > lmkd Tainted: G OE 6.18.21-android17-5-ga1a8e8cab9ec-4k > #1 PREEMPT 25372cd4750dcac3c0fe86b57d47c665f97a6046 > > [48916.569450] [9: lmkd: 536] pstate: 63402005 (nZCv daif > +PAN -UAO +TCO +DIT -SSBS BTYPE=--) > [48916.569452] [9: lmkd: 536] pc : > trace_user_fault_read+0x334/0x360 > [48916.569454] [9: lmkd: 536] lr : > trace_user_fault_read+0x330/0x360 > [48916.569456] [9: lmkd: 536] sp : ffffffc096993cc0 > [48916.569457] [9: lmkd: 536] x29: ffffffc096993cd0 x28: > 0000000000000065 x27: ffffff8817e45200 > [48916.569461] [9: lmkd: 536] x26: 0000000000000000 x25: > 000000011616c082 x24: 0000000000000000 > [48916.569463] [9: lmkd: 536] x23: 0000000000000009 x22: > 000000000000002b x21: ffffff880597f740 > [48916.569466] [9: lmkd: 536] x20: 0000007fc42dd940 x19: > ffffff8817e45770 x18: ffffffd5947ce240 > [48916.569468] [9: lmkd: 536] x17: 2073656972742079 x16: > 6e616d206f6f5420 x15: 3a726f727245205d > [48916.569471] [9: lmkd: 536] x14: 36333520203a646b x13: > 6563617073207265 x12: 0000000000000001 > [48916.569473] [9: lmkd: 536] x11: 6f74207365697274 x10: > 0000000000000001 x9 : 4bd9ad4516d75100 > [48916.569476] [9: lmkd: 536] x8 : 4bd9ad4516d75100 x7 : > 205d383431393635 x6 : 2e36313938345b0a > [48916.569479] [9: lmkd: 536] x5 : ffffffc080fa5998 x4 : > ffffffd591707202 x3 : 0001360a00000000 > [48916.569481] [9: lmkd: 536] x2 : ffffffc096993af4 x1 : > 00000000000000c0 x0 : 0000000000000028 > [48916.569484] [9: lmkd: 536] Call trace: > [48916.569486] [9: lmkd: 536] > trace_user_fault_read+0x334/0x360 (P) > [48916.569488] [9: lmkd: 536] tracing_mark_write+0x84/0x174 > [48916.569491] [9: lmkd: 536] __arm64_sys_write+0x2a0/0x5c0 > [48916.569494] [9: lmkd: 536] invoke_syscall+0x58/0xe4 > [48916.569498] [9: lmkd: 536] do_el0_svc+0x48/0xdc > [48916.569500] [9: lmkd: 536] el0_svc+0x3c/0x98 > [48916.569503] [9: lmkd: 536] > el0t_64_sync_handler+0x20/0x130 > [48916.569505] [9: lmkd: 536] el0t_64_sync+0x1c4/0x1c8 > [48916.569508] [9: lmkd: 536] Kernel panic - not syncing: > kernel: panic_on_warn set ... > > > The code at the WARN location mentioned in the log above is as follows. > > 7374 if (WARN_ONCE(trys++ > 100, "Error: Too many > tries to read user space")) > 7375 return NULL; > > > Our current analysis is as follows: > > In the Gmail process, during a low memory situation, LMKD writes strings > to /sys/kernel/tracing/trace_marker for systrace recording. At the same > time, it broadcasts a sigkill due to low memory, which is causing the > LMKD trace marker operation to stall. Hm, in my view, this warning indicates that the circuit breaker has triggered correctly, so that is not a bug. Under the heavy memory pressure and low-memory situation, the page can be reclaimed soon after it is copied. However, this seems a bit strange that we only checks the CPU-wide context switching in the loop. Instead, can we introduce a per-cpu sequence counter to per-cpu buffer, and check it? Could you try this ? >From f76d8e4400a5961725d17899f4290c9334987e2b Mon Sep 17 00:00:00 2001 From: Masami Hiramatsu <[email protected]> Date: Fri, 10 Jul 2026 12:11:00 +0900 Subject: [PATCH] tracing: Use per-CPU sequence counter in trace_user_fault_read trace_user_fault_read() copies trace data from user space to the per-CPU trace buffer. When preemption is enabled during the copy, it checks if any context switches occurred on the current CPU via nr_context_switches_cpu() to detect whether the buffer may have been corrupted by another trace writer. However, under heavy memory pressure, copying from user space can trigger page faults (e.g., for swapped-out BSS or anonymous pages) that block and cause a context switch. Because nr_context_switches_cpu() detects any context switch (even unrelated ones), it mistakenly assumes the buffer was corrupted. This leads to repeated retries (up to the 100-try limit), which causes a WARN_ONCE backtrace and returns -EFAULT to user space, even if no other task ever accessed the trace buffer. To mitigate this issue, replace the CPU-wide context switch check with a dedicated per-CPU sequence counter in struct trace_user_buf. Since only other tasks invoking trace_user_fault_read() on the same CPU will increment this counter, unrelated context switches (including those from page fault sleep) will no longer trigger retries. Signed-off-by: Masami Hiramatsu (Google) <[email protected]> --- kernel/trace/trace.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 1bc27c0ad029..46cec63f5798 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -5989,6 +5989,7 @@ static ssize_t write_marker_to_buffer(struct trace_array *tr, const char *buf, struct trace_user_buf { char *buf; + unsigned int sequence; }; static DEFINE_MUTEX(trace_user_buffer_mutex); @@ -6031,7 +6032,10 @@ static int user_fault_buffer_enable(struct trace_user_buf_info *tinfo, size_t si /* Clear each buffer in case of error */ for_each_possible_cpu(cpu) { - per_cpu_ptr(tinfo->tbuf, cpu)->buf = NULL; + struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu); + + tbuf->buf = NULL; + tbuf->sequence = 0; } for_each_possible_cpu(cpu) { @@ -6196,8 +6200,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo, trace_user_buf_copy copy_func, void *data) { int cpu = smp_processor_id(); - char *buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf; - unsigned int cnt; + struct trace_user_buf *tbuf = per_cpu_ptr(tinfo->tbuf, cpu); + char *buffer = tbuf->buf; + unsigned int seq; int trys = 0; int ret; @@ -6211,10 +6216,10 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo, return NULL; /* - * This acts similar to a seqcount. The per CPU context switches are + * This acts similar to a seqcount. The per CPU sequence counters are * recorded, migration is disabled and preemption is enabled. The * read of the user space memory is copied into the per CPU buffer. - * Preemption is disabled again, and if the per CPU context switches count + * Preemption is disabled again, and if the per CPU sequence count * is still the same, it means the buffer has not been corrupted. * If the count is different, it is assumed the buffer is corrupted * and reading must be tried again. @@ -6235,7 +6240,8 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo, preempt_enable_notrace(); preempt_disable_notrace(); cpu = smp_processor_id(); - buffer = per_cpu_ptr(tinfo->tbuf, cpu)->buf; + tbuf = per_cpu_ptr(tinfo->tbuf, cpu); + buffer = tbuf->buf; } /* @@ -6250,8 +6256,9 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo, if (WARN_ONCE(trys++ > 100, "Error: Too many tries to read user space")) return NULL; - /* Read the current CPU context switch counter */ - cnt = nr_context_switches_cpu(cpu); + /* Increment the per-CPU buffer sequence counter */ + tbuf->sequence++; + seq = tbuf->sequence; /* * Preemption is going to be enabled, but this task must @@ -6282,12 +6289,12 @@ char *trace_user_fault_read(struct trace_user_buf_info *tinfo, return NULL; /* - * Preemption is disabled again, now check the per CPU context - * switch counter. If it doesn't match, then another user space + * Preemption is disabled again, now check the per CPU sequence + * counter. If it doesn't match, then another user space * process may have schedule in and corrupted our buffer. In that * case the copying must be retried. */ - } while (nr_context_switches_cpu(cpu) != cnt); + } while (tbuf->sequence != seq); return buffer; } -- 2.43.0 -- Masami Hiramatsu (Google) <[email protected]>
