Public bug reported:

# Launchpad bug: D-state task (kopia) blocks RCU expedited grace period
→ podman crun hangs in synchronize_rcu_expedited during namespace
teardown; hung_task panic

**Summary:**
A task (kopia backup agent, PID 7741) wedged in uninterruptible sleep on CPU 11 
blocked an RCU expedited grace period for 62+ seconds (WARNING in 
`rcu_exp_handler`, kernel/rcu/tree_exp.h:808, then `rcu_preempt detected 
expedited stalls on CPUs/tasks: { 11-...D }`). Two concurrent `crun` (podman 
OCI runtime) processes doing mount-namespace teardown then hung for >122 s: one 
in `synchronize_rcu_expedited` from `namespace_unlock()` (`dissolve_on_fput`), 
the second blocked on a namespace mutex owned by the first. The system was 
configured with `kernel.hung_task_panic=1`, so khungtaskd panicked the kernel 
and kdump captured a full vmcore (available on request).

## Environment
- Ubuntu, linux 7.3.0-8-generic, PREEMPT(lazy)
- ASRock Z690 Steel Legend, Intel i7-14700K (24 CPUs), 48G RAM, zswap/zstd, 48G 
btrfs swapfile
- btrfs root (all subvolumes compress-force=zstd:1), CIFS mounts via systemd 
automount (no idle unmounts)
- Load at the time: podman containers with 6-second health checks (crun spawned 
continuously), kopia backup running in background, syncthing, LM Studio

## Timeline (uptime → wallclock: panic at 51117 s = 08:02)
```
50972 s  WARNING: kernel/rcu/tree_exp.h:808 at rcu_exp_handler+0x4a/0x190, 
CPU#11: kopia/7741
         (expedited-GP IPI lands on CPU 11 while a task there is in an 
unexpected state)
51035 s  rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 
11-...D } 62380 jiffies
         rcu: blocking rcu_node structures (internal RCU debug): l=1:0-13:0x800
         NMI backtrace of CPU 11 skipped: idling at intel_idle+0x62/0xd0
         (CPU is idle but its task remains in D state, still blocking the GP)
51117 s  INFO: task crun:1852547 blocked for more than 122 seconds.
           synchronize_rcu_expedited ← namespace_unlock ← dissolve_on_fput ← 
__fput (task work at syscall exit)
         INFO: task crun:1852551 blocked for more than 122 seconds.
           INFO: task crun:1852551 is blocked on a mutex likely owned by task 
crun:1852547.
         Kernel panic - not syncing: hung_task: blocked tasks   
[hung_task_panic=1, intentional]
```

## Deeper mechanism (why this is an RCU accounting bug, not slow IO)

Both RCU warnings caught the kopia task **running in user mode**
(userspace RIP 0x0033 in both traces):

```
47369 s  CPU 0, timer tick:  rcu_sched_clock_irq (tree_plugin.h:849) WARN, 
interrupted context: kopia @ RIP 0033 (userspace)
50972 s  CPU 11, expedited IPI: rcu_exp_handler (tree_exp.h:808) WARN, 
interrupted context: kopia @ RIP 0033 (userspace)
```

A task in userspace is by definition in a quiescent state — RCU should
have reported the QS for kopia's task at syscall exit
(`exit_to_user_mode` → deferred QS processing). Instead the deferred-QS
/ rcu_read_unlock_special state apparently remained set on the task
struct for **~1 hour** across two different reporting mechanisms (tick
and expedited IPI) that both noticed it and neither could clear it. When
kopia later blocked in D state on CPU 11, the CPU was marked `D` in the
GP mask and the expedited grace period could no longer complete at all —
hanging the crun namespace teardown that waited in
`synchronize_rcu_expedited`.

**Prodrome seen before on a different install:** the *same*
`rcu_sched_clock_irq (tree_plugin.h:849) CPU#: kopia/7741`-style warning
fired on 2026-10-02 15:00 (linux 7.3.0-6, previous filesystem/install,
same kopia binary), ~2 h before an unrelated VFS panic. The prodrome
reproduces across kernels 7.3.0-6/7.3.0-8 and across two filesystems,
always with kopia (Go runtime: many threads, dense short syscalls —
heavy user of the syscall-exit RCU exit path).

Suspect area: deferred quiescent-state processing for PREEMPT_RCU under
PREEMPT(lazy) — the special state set in the syscall path is not always
cleared on exit to userspace, so RCU waits for a QS from a task that is
a quiescent state.

## Notes
- Full kdump vmcore available (2.4 GB) — happy to provide or run crash(8) 
queries on request (task_struct.rcu_read_unlock_special / rnp->qsmask of the 
kopia task would confirm which flag was stuck).
- Possibly related to other compressed-writeback oopses seen on the same kernel 
(LP#2169415, LP#2169493, both in btrfs-delalloc workers with 
compress-force=zstd:1 on all subvolumes).
- The RCU machinery recovered nothing: the stall lasted until the deliberate 
panic; no self-healing within ~150 s.

## Attachments
- /var/crash/202610050803/dmesg.202610050803 (kdump vmcore-dmesg with all stack 
traces, collected by apport)

## Request
Please route to RCU maintainers (Cc: Paul McKenney) — the rcu_exp_handler 
WARNING at tree_exp.h:808 plus a CPU whose task sits in D state blocking 
expedited GPs indefinitely looks like an RCU-side accounting/state issue rather 
than a one-off IO stall.

ProblemType: Bug
DistroRelease: Ubuntu 26.10
Package: linux-image-7.3.0-8-generic 7.3.0-8.8
ProcVersionSignature: Ubuntu 7.3.0-8.8-generic 7.3.0-rc5
Uname: Linux 7.3.0-8-generic x86_64
ApportVersion: 2.36.0-0ubuntu1
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: KDE
Date: Mon Oct  5 13:06:52 2026
InstallationDate: Installed on 2026-10-03 (2 days ago)
InstallationMedia: Kubuntu 26.10 "Stonking Stingray" - Beta amd64 (20260929)
IwDevWlp3s0f0Link: Not connected.
MachineType: ASRock Z690 Steel Legend
ProcFB: 0 nvidia-drmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-7.3.0-8-generic 
root=UUID=984a3dfc-b31b-4c9e-b1fd-19f0c4b38e9b ro rootflags=subvol=@ quiet 
splash 
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No 
PulseAudio daemon running, or not running as session daemon.
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 08/09/2024
dmi.bios.release: 5.27
dmi.bios.vendor: American Megatrends International, LLC.
dmi.bios.version: 19.02
dmi.board.name: Z690 Steel Legend
dmi.board.vendor: ASRock
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: 
dmi:bvnAmericanMegatrendsInternational,LLC.:bvr19.02:bd08/09/2024:br5.27:svnASRock:pnZ690SteelLegend:pvrToBeFilledByO.E.M.:rvnASRock:rnZ690SteelLegend:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:skuToBeFilledByO.E.M.:pfaToBeFilledByO.E.M.:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: Z690 Steel Legend
dmi.product.sku: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: ASRock

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug stonking wayland-session

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169559

Title:
  D-state task (kopia) blocks RCU expedited grace period → podman crun
  hangs in synchronize_rcu_expedited during namespace teardown;
  hung_task panic

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2169559/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to