Public bug reported:
Since installing Ubuntu 26.04 the machine spontaneously reboots (kdump catches
a kernel
Oops) a few minutes after starting to download several files with the Firefox
snap while
snapd AppArmor prompting (experimental.apparmor-prompting=true,
prompting-client snap) is
enabled. It is reproducible: downloading 4 PDFs in a row was usually enough.
Every crash follows the same pattern:
1. `WARNING: kernel/workqueue.c:2351 at __queue_work.part.0+0x190/0x390`, fired
from
`delayed_work_timer_fn` -> `__run_timers` -> `run_timer_softirq` on idle CPUs
(swapper), often on two CPUs within the same microsecond.
2. About 15 s later:
`Oops: general protection fault, probably for non-canonical address
0xdead00000000012a`
with `RIP: __run_timers+0x1e0/0x2c0` (walking a timer whose entry.next is
LIST_POISON2).
### Root cause analysis (crash + linux-image-7.0.0-38-generic-dbgsym)
I opened two vmcores (202610221536 and 202610221601) with `crash`:
- In both dumps, the `work_struct` reported by every WARNING (RBX in the
`__queue_work` frame) is a delayed_work embedded at offset 0x100 of an object
from the
**`apparmor_auditcache`** slab, with `func = audit_cache_work_function`
(security/apparmor/notify.c:845).
- The `timer_list` that `__run_timers` was walking at the GPF (R13) is the
timer of a
delayed_work in the same `apparmor_auditcache` slab (object + 0x120), with
`function = delayed_work_timer_fn` and `entry.next = 0xdead000000000122`
(LIST_POISON2).
- When the WARNING fires, the work_structs have `data = 0` and `entry.next =
NULL`. It
looks like the audit cache entry is re-initialized or freed and reused while
its
delayed_work timer is still pending (for example, the entry is freed without
`cancel_delayed_work_sync()`), so the same timer is detached or enqueued
twice.
Example from dump 202610221601:
```
crash> struct work_struct ffff8e9bde2a2298
data = { counter = 0 }, entry = { next = 0x0, prev = 0xffff8e9bde2a22a0 },
func = 0xffffffffb33c1ab0 <audit_cache_work_function>
crash> kmem ffff8e9bde2a2298
ffff8e9ac04a2b00 344 168 188 4 16k
apparmor_auditcache
[ffff8e9bde2a2198]
crash> struct timer_list ffff8e9cc94a76e0 # R13 at the GPF
entry = { next = 0xdead000000000122, pprev = 0xffffced4c05a0ea8 },
function = 0xffffffffb2c71a90 <delayed_work_timer_fn>
crash> kmem ffff8e9cc94a76e0
ffff8e9ac04a2b00 344 168 188 4 16k
apparmor_auditcache
[ffff8e9cc94a75c0]
```
### Things ruled out
- **NVIDIA driver:** the same WARNING occurs on 7.0.0-34 without the NVIDIA
modules loaded.
The kernel is tainted (O) only because of nvidia 595.99.02 (open).
- **Timer migration / NOHZ:** the crash still happens with
`kernel.timer_migration=0` and
with `nohz=off`.
- **Memory pressure / OOM:** none.
### Workaround
`sudo snap set system experimental.apparmor-prompting=false`
After this change, 11 PDFs downloaded in parallel produced no WARNING and no
crash. Before
the change, the first WARNING appeared about 3 minutes after boot.
### Steps to reproduce
1. Enable snapd AppArmor prompting (prompting-client installed, home access
prompting on).
2. In the Firefox snap, download several PDFs into ~/Downloads in quick
succession.
3. Within a few minutes, `workqueue.c:2351` WARNINGs appear in dmesg, followed
by a GPF in
`__run_timers` and a reboot.
## System information
- Ubuntu 26.04.1, GNOME Wayland
- Kernel 7.0.0-38-generic (7.0.0-38.38, upstream 7.0.14). Also reproduced on
7.0.0-34.
- snapd 2.77.1+ubuntu26.04.2, prompting-client 0+git.dd8c750 (rev 228), firefox
snap 155.0.1-1 (rev 8863)
- ASUS TUF Gaming F16 FX608JHR, BIOS FX608JHR.312, Intel i7-14650HX, Intel UHD
+ NVIDIA RTX 5050 Laptop
## Attachments
- launchpad-dmesg-202610221601.txt: WARNINGs and Oops from the 16:01 crash
- launchpad-dmesg-202610221536.txt: WARNINGs and Oops from the 15:36 crash
- Full vmcores (~400-460 MB each) are available on request.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Tags: apparmor kernel-oops resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172038
Title:
AppArmor notify audit cache: delayed_work timer corruption causes
"workqueue.c:2351" WARNINGs and GPF in __run_timers
(0xdead00000000012a) when apparmor-prompting is enabled
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2172038/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
