Hi Salvatore,
sorry, I somehow missed your mail.
Am 21.03.25 um 21:07 schrieb Salvatore Bonaccorso:
While we had no followup after your last email, I wonder now if you
still experience the issues with the most recent 6.1.y kernel
available in Debian (6.1.128-1).
Can you report back? Can you test on that machine as well newer
kernels ideally from unstable (6.12.16-1) or experimental
(6.13.4-1~exp1)?
Any news here?
With 6.1.128-1, there are still backtraces, though I don't experience
any slowness. I could only retrieve them from dmesg output as the kernel
logfiles containing them are already rotated. Also, I removed noise
unrelated to to the crash. See attachment.
I'll install a newer kernel tomorrow and let you know the outcome.
Willi
[1478620.041716] rcu_gp_fqs_loop+0x141/0x550
[1478620.041719] rcu_gp_kthread+0xd4/0x190
[1478620.041721] kthread+0xda/0x100
[1478620.041725] ? kthread_complete_and_exit+0x20/0x20
[1478620.041727] ret_from_fork+0x22/0x30
[1478620.041732] </TASK>
[1478620.041733] rcu: Stack dump where RCU GP kthread last ran:
[1478620.042159] CPU: 1 PID: 746 Comm: nginx Tainted: G L
6.1.0-28-amd64 #1 Debian 6.1.119-1
[1478620.042163] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1478620.042164] RIP: 0010:exit_to_user_mode_prepare+0xab/0x1e0
[1478620.042169] Code: 05 3a 57 ad 4d 48 8b 15 fb 72 e8 02 89 c0 48 0f a3 02 0f
82 25 01 00 00 48 8b 5d 00 f7 c3 0e 30 02 00 74 81 fb 0f 1f 44 00 00 <f6> c3 08
0f 85 00 01 00 00 f6 c7 10 0f 85 ea 00 00 00 f6 c7 20 0f
[1478620.042170] RSP: 0000:ffff9ffcc0cf3f30 EFLAGS: 00000202
[1478620.045530] RAX: 0000000000000008 RBX: 0000000000000008 RCX:
000000000000080b
[1478620.045533] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
ffff9ffcc0cf3f58
[1478620.045534] RBP: ffff920f03586600 R08: 0000000000000000 R09:
0000000000000000
[1478620.045535] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff9ffcc0cf3f58
[1478620.045536] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[1478620.045539] FS: 00007fe2ee3ae740(0000) GS:ffff920fb9d00000(0000)
knlGS:0000000000000000
[1478620.045541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1478620.045542] CR2: 00007f8de5a2d1a0 CR3: 0000000105b3e000 CR4:
00000000003506e0
[1478620.045545] Call Trace:
[1478620.045585] <IRQ>
[1478620.045590] ? rcu_check_gp_kthread_starvation+0xec/0xfd
[1478620.045597] ? rcu_sched_clock_irq.cold+0x2ce/0x459
[1478620.045600] ? update_process_times+0x70/0xb0
[1478620.045605] ? tick_sched_handle+0x22/0x60
[1478620.045608] ? tick_sched_timer+0x63/0x80
[1478620.045609] ? tick_sched_do_timer+0xa0/0xa0
[1478620.045611] ? __hrtimer_run_queues+0x112/0x2b0
[1478620.045613] ? hrtimer_interrupt+0xf4/0x210
[1478620.045616] ? __sysvec_apic_timer_interrupt+0x5d/0x110
[1478620.045620] ? sysvec_apic_timer_interrupt+0x69/0x90
[1478620.045623] </IRQ>
[1478620.045624] <TASK>
[1478620.045625] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[1478620.045630] ? exit_to_user_mode_prepare+0xab/0x1e0
[1478620.045633] irqentry_exit_to_user_mode+0x5/0x30
[1478620.045657] asm_sysvec_reschedule_ipi+0x16/0x20
[1478620.045660] RIP: 0033:0x7fe2ee02d3e7
[1478620.045816] Code: 89 04 24 48 89 c5 48 f7 f1 8b 43 20 48 39 c2 73 0b 8b 4b
1c 48 89 e8 31 d2 48 f7 f1 48 8b 03 4c 8b 73 08 48 63 d2 4c 8d 24 d0 <49> 8b 1c
24 48 85 db 74 20 48 39 6b 10 75 0d 48 8b 3b 4c 89 ee 41
[1478620.045818] RSP: 002b:00007fff72c34120 EFLAGS: 00000212
[1478620.045820] RAX: 000055d874f1e300 RBX: 000055d874f14b10 RCX:
0000000000000400
[1478620.045821] RDX: 00000000000000a2 RSI: 00007fff72c34170 RDI:
0000000000000000
[1478620.045822] RBP: 000000000a0000a2 R08: 0000000000000000 R09:
0000000000000000
[1478620.045823] R10: 0000000000000000 R11: 0000000000000000 R12:
000055d874f1e810
[1478620.045824] R13: 00007fff72c34170 R14: 00007fe2edfe82f0 R15:
00007fff72c3436c
[1478620.045826] </TASK>
[1553669.664319] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[1553671.007041] rcu: 1-...0: (11 ticks this GP)
idle=5c24/1/0x4000000000000000 softirq=5272915/5272917 fqs=2559
[1553671.010934] (detected by 0, t=5252 jiffies, g=9373849, q=55 ncpus=2)
[1553671.010980] Sending NMI from CPU 0 to CPUs 1:
[1553671.328614] NMI backtrace for cpu 1
[1553671.328635] CPU: 1 PID: 325216 Comm: kworker/u4:0 Tainted: G L
6.1.0-28-amd64 #1 Debian 6.1.119-1
[1553671.328653] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1553671.328671] Workqueue: writeback wb_workfn (flush-254:0)
[1553671.329988] RIP: 0010:asm_sysvec_reschedule_ipi+0x0/0x20
[1553671.330447] Code: e9 15 07 00 00 0f 1f 44 00 00 0f 01 ca fc 6a ff e8 d5 05
00 00 48 89 c4 48 89 e7 e8 fa f9 e2 ff e9 f5 06 00 00 0f 1f 44 00 00 <0f> 01 ca
fc 6a ff e8 b5 05 00 00 48 89 c4 48 89 e7 e8 fa 1b e3 ff
[1553671.330450] RSP: 0018:ffff9ffcc05a78d8 EFLAGS: 00000046
[1553671.330452] RAX: ffff920f09ce4fc8 RBX: ffff920f09ce4f80 RCX:
0000000000000017
[1553671.330467] RDX: 000018db4ac5b01e RSI: 0000000000000000 RDI:
0005d933724671ce
[1553671.330468] RBP: ffff9ffcc05a7930 R08: 0006e3a9ac5b3dce R09:
0000000000000001
[1553671.330469] R10: 0000000000037880 R11: ffff920fbffd5c00 R12:
0000000000000000
[1553671.330470] R13: ffff9ffcc05a7d80 R14: 0000000000000000 R15:
ffff920f9ff76b60
[1553671.330472] FS: 0000000000000000(0000) GS:ffff920fb9d00000(0000)
knlGS:0000000000000000
[1553671.330473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1553671.330474] CR2: 0000558dc9201a27 CR3: 000000010528e000 CR4:
00000000003506e0
[1553671.330489] Call Trace:
[1553671.330639] <TASK>
[1553671.330640] RIP: 0010:blk_mq_rq_ctx_init.isra.0+0xb1/0x160
[1553671.330998] Code: 00 00 00 00 31 c0 48 c7 43 58 00 00 00 00 66 89 43 7a 48
8d 43 48 48 89 43 48 4c 89 63 60 48 89 43 50 48 c7 43 70 00 00 00 00 <48> c7 83
90 00 00 00 00 00 00 00 c7 43 7c 00 00 00 00 48 c7 83 00
[1553671.331000] RSP: 0018:ffff9ffcc05a7908 EFLAGS: 00000246
[1553671.331704] ? blk_mq_submit_bio+0x1b7/0x610
[1553671.331711] ? submit_bio_noacct_nocheck+0x32e/0x370
[1553671.331822] ? submit_bio_noacct+0x79/0x4b0
[1553671.331824] ? __block_write_full_page+0x1f9/0x4e0
[1553671.331839] ? __find_get_block+0x330/0x330
[1553671.331842] ? blkdev_write_begin+0x20/0x20
[1553671.331861] ? __writepage+0x19/0x70
[1553671.332004] ? write_cache_pages+0x183/0x4c0
[1553671.332005] ? dirty_background_bytes_handler+0x60/0x60
[1553671.332117] ? generic_writepages+0x7b/0xb0
[1553671.332121] ? do_writepages+0xc0/0x1c0
[1553671.332122] ? _raw_spin_unlock_irqrestore+0xa/0x40
[1553671.332125] ? __percpu_counter_sum+0x5a/0x70
[1553671.332444] ? __writeback_single_inode+0x3d/0x360
[1553671.332549] ? fprop_fraction_percpu+0x2b/0x80
[1553671.332578] ? writeback_sb_inodes+0x1f5/0x4d0
[1553671.332581] ? __writeback_inodes_wb+0x4c/0xf0
[1553671.332583] ? wb_writeback+0x204/0x2f0
[1553671.332585] ? wb_workfn+0x350/0x4f0
[1553671.332587] ? process_one_work+0x1c7/0x380
[1553671.332648] ? worker_thread+0x4d/0x380
[1553671.332651] ? rescuer_thread+0x3a0/0x3a0
[1553671.332653] ? kthread+0xda/0x100
[1553671.332655] ? kthread_complete_and_exit+0x20/0x20
[1553671.332657] ? ret_from_fork+0x22/0x30
[1553671.332688] </TASK>
[1553786.640316] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[1553786.640881] rcu: 1-...0: (1 GPs behind) idle=f3c4/1/0x4000000000000000
softirq=5273270/5273271 fqs=1026
[1553786.641294] (detected by 0, t=5254 jiffies, g=9374541, q=31 ncpus=2)
[1553786.641298] Sending NMI from CPU 0 to CPUs 1:
[1553816.599715] NMI backtrace for cpu 1
[1553816.599722] CPU: 1 PID: 325431 Comm: postgres Tainted: G L
6.1.0-28-amd64 #1 Debian 6.1.119-1
[1553816.599727] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1553816.599729] RIP: 0010:asm_sysvec_reschedule_ipi+0x0/0x20
[1553816.606236] Code: e9 15 07 00 00 0f 1f 44 00 00 0f 01 ca fc 6a ff e8 d5 05
00 00 48 89 c4 48 89 e7 e8 fa f9 e2 ff e9 f5 06 00 00 0f 1f 44 00 00 <0f> 01 ca
fc 6a ff e8 b5 05 00 00 48 89 c4 48 89 e7 e8 fa 1b e3 ff
[1553816.606843] RSP: 0000:ffff9ffcc0fc3c18 EFLAGS: 00000046
[1553816.606846] RAX: 0000000000000000 RBX: 0000000000000001 RCX:
0000000000000200
[1553816.606847] RDX: ffffd513842421c0 RSI: ffffd51384242200 RDI:
ffff920f09087000
[1553816.606848] RBP: ffff920fb9d378c0 R08: 0000000000000000 R09:
000000000008c62c
[1553816.606849] R10: ffff9ffcc0fc3d48 R11: ffff920fb9d378d8 R12:
0000000000000000
[1553816.606849] R13: ffff920fbffd5c00 R14: ffff920fbffd6e00 R15:
ffffd513842421c0
[1553816.606852] FS: 00007fbe29023a80(0000) GS:ffff920fb9d00000(0000)
knlGS:0000000000000000
[1553816.606854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1553816.606854] CR2: 00007fbe29705d90 CR3: 0000000108f6e000 CR4:
00000000003506e0
[1553816.606858] Call Trace:
[1553816.606861] <TASK>
[1553816.606861] RIP: 0010:clear_page_rep+0x7/0x10
[1553816.606886] Code: fd 48 89 df 48 8b 5b 08 48 83 c7 18 e8 52 1c ff ff 48 89
5d 20 48 85 db 75 e7 5b 5d e9 82 0f 42 00 cc cc b9 00 02 00 00 31 c0 <f3> 48 ab
e9 71 0f 42 00 90 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00
[1553816.606887] RSP: 0000:ffff9ffcc0fc3c48 EFLAGS: 00000246
[1553816.606889] ? get_page_from_freelist+0xe24/0x11b0
[1553816.607315] ? __mod_memcg_lruvec_state+0x41/0x90
[1553816.607725] ? do_set_pte+0x189/0x1d0
[1553816.608200] ? __alloc_pages+0x1dc/0x330
[1553816.608203] ? pte_alloc_one+0x13/0x40
[1553816.608232] ? do_fault+0x3a6/0x410
[1553816.608234] ? __handle_mm_fault+0x660/0xfa0
[1553816.608238] ? handle_mm_fault+0xdb/0x2d0
[1553816.608240] ? do_user_addr_fault+0x191/0x550
[1553816.608242] ? exc_page_fault+0x70/0x170
[1553816.706320] ? asm_exc_page_fault+0x22/0x30
[1553816.706341] </TASK>
[1553898.981502] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[1553899.032494] rcu: 1-...0: (17 ticks this GP)
idle=26dc/1/0x4000000000000000 softirq=5273460/5273460 fqs=2473
[1553899.080576] (detected by 0, t=5253 jiffies, g=9374905, q=180
ncpus=2)
[1553899.080584] Sending NMI from CPU 0 to CPUs 1:
[1553930.707039] NMI backtrace for cpu 1
[1553930.707046] CPU: 1 PID: 125321 Comm: systemd-journal Tainted: G
L 6.1.0-28-amd64 #1 Debian 6.1.119-1
[1553930.707050] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1553930.707052] RIP: 0010:asm_sysvec_reschedule_ipi+0x0/0x20
[1553930.707060] Code: e9 15 07 00 00 0f 1f 44 00 00 0f 01 ca fc 6a ff e8 d5 05
00 00 48 89 c4 48 89 e7 e8 fa f9 e2 ff e9 f5 06 00 00 0f 1f 44 00 00 <0f> 01 ca
fc 6a ff e8 b5 05 00 00 48 89 c4 48 89 e7 e8 fa 1b e3 ff
[1553930.707062] RSP: 0000:fffffe000003dfd8 EFLAGS: 00000002
[1553930.707064] RAX: 0000000000000001 RBX: 0000000001d83220 RCX:
0000000001d83220
[1553930.707065] RDX: 0000000000000001 RSI: 00007ff4a7583220 RDI:
00005586b9e9d2d0
[1553930.707066] RBP: 00005586b9e9d2d0 R08: 00005586b9eaa2f0 R09:
00005586b9e9d308
[1553930.707067] R10: 0000000000000003 R11: 00007fffd813ad60 R12:
00007fffd813a7a0
[1553930.707068] R13: 00007fffd813a700 R14: 0000000000000001 R15:
00007ff4a7583220
[1553930.707071] FS: 00007ff4b513c5c0(0000) GS:ffff920fb9d00000(0000)
knlGS:ffff920fb9d00000
[1553930.707072] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1553930.707073] CR2: 00007ff4ac25f9b8 CR3: 00000001053f6000 CR4:
00000000003506e0
[1553930.707077] Call Trace:
[1554075.235420] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[1554075.236024] rcu: 1-...0: (2 ticks this GP) idle=d444/0/0x1
softirq=5274605/5274606 fqs=2422
[1554075.241385] (detected by 0, t=5252 jiffies, g=9376949, q=41 ncpus=2)
[1554075.241391] Sending NMI from CPU 0 to CPUs 1:
[1554078.320310] NMI backtrace for cpu 1
[1554078.320355] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G L
6.1.0-28-amd64 #1 Debian 6.1.119-1
[1554078.320359] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1554078.320360] RIP: 0010:asm_sysvec_reschedule_ipi+0x0/0x20
[1554078.320369] Code: e9 15 07 00 00 0f 1f 44 00 00 0f 01 ca fc 6a ff e8 d5 05
00 00 48 89 c4 48 89 e7 e8 fa f9 e2 ff e9 f5 06 00 00 0f 1f 44 00 00 <0f> 01 ca
fc 6a ff e8 b5 05 00 00 48 89 c4 48 89 e7 e8 fa 1b e3 ff
[1554078.321396] RSP: 0018:ffff9ffcc00d4c38 EFLAGS: 00000046
[1554078.321398] RAX: 0000000000000000 RBX: 0000000000000001 RCX:
0000000000000200
[1554078.321400] RDX: ffffd5138425a6c0 RSI: ffffd5138425a800 RDI:
ffff920f0969b000
[1554078.321400] RBP: ffff920fb9d378c0 R08: 0000000000000000 R09:
ffffd513855d1008
[1554078.321401] R10: 0000000000000002 R11: ffff920fb9d37968 R12:
0000000000000003
[1554078.321402] R13: ffff920fbffd5c00 R14: ffff920fbffd6e00 R15:
ffffd5138425a600
[1554078.321405] FS: 0000000000000000(0000) GS:ffff920fb9d00000(0000)
knlGS:0000000000000000
[1554078.321407] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1554078.321408] CR2: 00007fdea74a3024 CR3: 0000000116584000 CR4:
00000000003506e0
[1554078.321411] Call Trace:
[1554078.321413] <IRQ>
[1554078.324721] RIP: 0010:clear_page_rep+0x7/0x10
[1554078.324847] Code: fd 48 89 df 48 8b 5b 08 48 83 c7 18 e8 52 1c ff ff 48 89
5d 20 48 85 db 75 e7 5b 5d e9 82 0f 42 00 cc cc b9 00 02 00 00 31 c0 <f3> 48 ab
e9 71 0f 42 00 90 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00
[1554078.324849] RSP: 0018:ffff9ffcc00d4c68 EFLAGS: 00000246
[1554078.324852] ? get_page_from_freelist+0xe24/0x11b0
[1554078.324861] ? __napi_alloc_skb+0xd9/0x1c0
[1554078.324918] ? __alloc_pages+0x1dc/0x330
[1554078.324920] ? skb_page_frag_refill+0x6a/0x120
[1554078.325280] ? try_fill_recv+0x301/0x7f0 [virtio_net]
[1554078.325397] ? napi_gro_receive+0x91/0x200
[1554078.325426] ? virtnet_poll+0x347/0x4c3 [virtio_net]
[1554078.325432] ? __napi_poll+0x2b/0x160
[1554078.325447] ? net_rx_action+0x29e/0x350
[1554078.325449] ? __napi_schedule+0x84/0xa0
[1554078.325452] ? handle_softirqs+0xd7/0x280
[1554078.325479] ? handle_edge_irq+0x87/0x220
[1554078.325536] ? __irq_exit_rcu+0xac/0xe0
[1554078.325538] ? common_interrupt+0x82/0xa0
[1554078.325541] </IRQ>
[1554078.325543] <TASK>
[1554078.325544] ? asm_common_interrupt+0x22/0x40
[1554078.325547] ? __sched_text_end+0x6/0x6
[1554078.325550] ? native_safe_halt+0xb/0x10
[1554078.325551] ? default_idle+0xa/0x10
[1554078.325552] ? default_idle_call+0x38/0xf0
[1554078.325554] ? do_idle+0x21b/0x2a0
[1554078.325583] ? cpu_startup_entry+0x26/0x30
[1554078.325586] ? start_secondary+0x12a/0x150
[1554078.325619] ? secondary_startup_64_no_verify+0xe5/0xeb
[1554078.325624] </TASK>
[1697798.122028] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[1697798.182030] rcu: 1-...!: (12 ticks this GP)
idle=0344/1/0x4000000000000000 softirq=5767672/5767674 fqs=244
[1697798.185216] (detected by 0, t=8802 jiffies, g=10238565, q=24
ncpus=2)
[1697798.185695] Sending NMI from CPU 0 to CPUs 1:
[1697798.718336] NMI backtrace for cpu 1
[1697798.720081] CPU: 1 PID: 125321 Comm: systemd-journal Tainted: G
L 6.1.0-28-amd64 #1 Debian 6.1.119-1
[1697798.720107] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1697798.720929] RIP: 0010:___slab_alloc+0x474/0x9a0
[1697798.734783] Code: 76 10 48 85 f6 0f 85 15 05 00 00 48 8b 44 24 48 49 89 d8
49 89 46 10 8b 55 28 4c 01 c2 48 89 d0 48 0f c8 48 33 85 b8 00 00 00 <48> 33 02
49 89 06 49 81 46 08 00 20 00 00 41 f7 c1 00 02 00 00 74
[1697798.734786] RSP: 0018:ffff9ffcc6157a58 EFLAGS: 00000082
[1697798.734801] RAX: a20b44ddea6d9f1a RBX: 0000000000000000 RCX:
0000000080150015
[1697798.734814] RDX: ffff920f03d77360 RSI: ffffd513840f5dc0 RDI:
0000000000000200
[1697798.734815] RBP: ffff920f00042800 R08: ffff920f03d77300 R09:
0000000000000246
[1697798.734816] R10: 0000000000000082 R11: 0000000011000088 R12:
00000000ffffffff
[1697798.734817] R13: 0000000000000dc0 R14: ffff920fb9d36100 R15:
ffffffffc06cd405
[1697798.734823] FS: 00007ff4b513c5c0(0000) GS:ffff920fb9d00000(0000)
knlGS:0000000000000000
[1697798.734824] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1697798.734825] CR2: 00007fad7f65ea28 CR3: 00000001053f6000 CR4:
00000000003506e0
[1697798.735076] Call Trace:
[1697798.737511] <NMI>
[1697798.741983] ? nmi_cpu_backtrace.cold+0x1c/0x79
[1697798.749230] ? nmi_cpu_backtrace_handler+0xd/0x20
[1697798.756062] ? nmi_handle+0x5d/0x120
[1697798.758630] ? default_do_nmi+0x40/0x130
[1697798.758797] ? exc_nmi+0x11e/0x150
[1697798.758799] ? end_repeat_nmi+0x16/0x67
[1697798.761442] ? ext4_htree_store_dirent+0x35/0x110 [ext4]
[1697798.778131] ? ___slab_alloc+0x474/0x9a0
[1697798.779738] ? ___slab_alloc+0x474/0x9a0
[1697798.779741] ? ___slab_alloc+0x474/0x9a0
[1697798.779744] </NMI>
[1697798.779768] <TASK>
[1697798.779772] ? ext4_htree_store_dirent+0x35/0x110 [ext4]
[1697798.781413] ? pagecache_get_page+0x36/0x60
[1697798.785631] ? _raw_spin_unlock+0xa/0x30
[1697798.785636] ? __getblk_gfp+0x2c/0x370
[1697798.795137] ? ext4_sync_file+0x370/0x370 [ext4]
[1697798.795181] __kmem_cache_alloc_node+0x110/0x2a0
[1697798.795668] ? ext4_htree_store_dirent+0x35/0x110 [ext4]
[1697798.797602] ? ext4_htree_store_dirent+0x35/0x110 [ext4]
[1697798.797634] __kmalloc+0x49/0x150
[1697798.800340] ext4_htree_store_dirent+0x35/0x110 [ext4]
[1697798.805650] htree_dirblock_to_tree+0x1f2/0x370 [ext4]
[1697798.810609] ext4_htree_fill_tree+0x239/0x3d0 [ext4]
[1697798.810651] ext4_readdir+0x78d/0xb60 [ext4]
[1697798.812803] iterate_dir+0x17a/0x1c0
[1697798.814326] __x64_sys_getdents64+0x84/0x120
[1697798.817912] ? compat_filldir+0x190/0x190
[1697798.817920] do_syscall_64+0x55/0xb0
[1697798.820146] ? exit_to_user_mode_prepare+0x40/0x1e0
[1697798.829770] ? syscall_exit_to_user_mode+0x1e/0x40
[1697798.829786] ? do_syscall_64+0x61/0xb0
[1697798.829788] ? do_syscall_64+0x61/0xb0
[1697798.829790] ? do_syscall_64+0x61/0xb0
[1697798.829792] ? syscall_exit_to_user_mode+0x1e/0x40
[1697798.829794] ? do_syscall_64+0x61/0xb0
[1697798.829796] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[1697798.829863] RIP: 0033:0x7ff4b4cef003
[1697798.829908] Code: df 66 0f 1f 44 00 00 48 89 ef e8 78 61 fb ff 4c 89 e0 5b
5d 41 5c c3 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00
f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 c9 1d 10 00 f7 d8
[1697798.829910] RSP: 002b:00007fffd813aa18 EFLAGS: 00000293 ORIG_RAX:
00000000000000d9
[1697798.829913] RAX: ffffffffffffffda RBX: 00005586b9ed51e0 RCX:
00007ff4b4cef003
[1697798.829915] RDX: 0000000000008000 RSI: 00005586b9ed51e0 RDI:
0000000000000021
[1697798.829915] RBP: 00005586b9ed51b4 R08: 00007ff4b4df22e0 R09:
00005586b9f0ea90
[1697798.829916] R10: 0000000000000000 R11: 0000000000000293 R12:
fffffffffffffdc8
[1697798.829917] R13: 0000000000000000 R14: 00005586b9ed51b0 R15:
00005586b81bfca3
[1697798.829921] </TASK>
[1697798.831002] rcu: rcu_preempt kthread starved for 8302 jiffies! g10238565
f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
[1697798.835343] rcu: Unless rcu_preempt kthread gets sufficient CPU time,
OOM is now expected behavior.
[1697798.836163] rcu: RCU grace-period kthread stack dump:
[1697798.837866] task:rcu_preempt state:R running task stack:0
pid:15 ppid:2 flags:0x00004000
[1697798.837887] Call Trace:
[1697798.839070] <TASK>
[1697798.839076] __schedule+0x34d/0x9e0
[1697798.840190] ? rcu_gp_cleanup+0x460/0x460
[1697798.840216] schedule+0x5a/0xd0
[1697798.840220] schedule_timeout+0x94/0x150
[1697798.840222] ? __bpf_trace_tick_stop+0x10/0x10
[1697798.840249] rcu_gp_fqs_loop+0x141/0x550
[1697798.840589] rcu_gp_kthread+0xd4/0x190
[1697798.840595] kthread+0xda/0x100
[1697798.841114] ? kthread_complete_and_exit+0x20/0x20
[1697798.841119] ret_from_fork+0x22/0x30
[1697798.841373] </TASK>
[1697798.841376] rcu: Stack dump where RCU GP kthread last ran:
[1697798.842299] CPU: 0 PID: 357366 Comm: kworker/u4:1 Tainted: G L
6.1.0-28-amd64 #1 Debian 6.1.119-1
[1697798.842304] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1697798.845058] Workqueue: writeback wb_workfn (flush-254:0)
[1697798.845092] RIP: 0010:blk_mq_rq_ctx_init.isra.0+0x25/0x160
[1697798.845323] Code: 55 00 0f 1f 00 0f 1f 44 00 00 41 54 41 89 d0 49 89 cc 55
48 89 fd 48 89 f7 53 48 8b 45 30 48 8b 75 28 48 8b 4d 00 4a 8b 1c c7 <48> 89 0b
48 89 73 08 48 89 43 10 8b 45 10 89 43 18 f6 45 08 04 8b
[1697798.845325] RSP: 0018:ffff9ffcc293f8e8 EFLAGS: 00000282
[1697798.845327] RAX: ffff920f008eb600 RBX: ffff920f09c72e80 RCX:
ffff920f10b5a0e8
[1697798.845328] RDX: 000000000000001f RSI: ffffbffcbfc031c0 RDI:
ffff920f10bfb000
[1697798.845329] RBP: ffff9ffcc293f910 R08: 000000000000001f R09:
ffff920f033c8000
[1697798.845330] R10: 0000000000037880 R11: ffff920fbffd5c00 R12:
0000000000000000
[1697798.845331] R13: ffff9ffcc293fd80 R14: 0000000000000000 R15:
ffff920f09c0d000
[1697798.845335] FS: 0000000000000000(0000) GS:ffff920fb9c00000(0000)
knlGS:0000000000000000
[1697798.845336] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1697798.845337] CR2: 000055e9e46060b0 CR3: 00000001b9d84000 CR4:
00000000003506f0
[1697798.845341] Call Trace:
[1697798.845489] <IRQ>
[1697798.845508] ? rcu_check_gp_kthread_starvation+0xec/0xfd
[1697798.845622] ? rcu_sched_clock_irq.cold+0x2ce/0x459
[1697798.845626] ? update_process_times+0x70/0xb0
[1697798.846455] ? tick_sched_handle+0x22/0x60
[1697798.846500] ? tick_sched_timer+0x63/0x80
[1697798.846501] ? tick_sched_do_timer+0xa0/0xa0
[1697798.846503] ? __hrtimer_run_queues+0x112/0x2b0
[1697798.846506] ? hrtimer_interrupt+0xf4/0x210
[1697798.846508] ? __sysvec_apic_timer_interrupt+0x5d/0x110
[1697798.846621] ? sysvec_apic_timer_interrupt+0x69/0x90
[1697798.846625] </IRQ>
[1697798.846626] <TASK>
[1697798.846627] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[1697798.846632] ? blk_mq_rq_ctx_init.isra.0+0x25/0x160
[1697798.846634] blk_mq_submit_bio+0x1b7/0x610
[1697798.846999] submit_bio_noacct_nocheck+0x32e/0x370
[1697798.847036] ? submit_bio_noacct+0x79/0x4b0
[1697798.847041] ext4_io_submit+0x20/0x40 [ext4]
[1697798.847192] ext4_writepages+0x470/0x11e0 [ext4]
[1697798.849019] ? _raw_spin_unlock+0xa/0x30
[1697798.849028] ? wb_update_bandwidth+0x4e/0x70
[1697798.849032] do_writepages+0xc0/0x1c0
[1697798.849063] ? __wb_calc_thresh+0x4b/0x140
[1697798.849065] __writeback_single_inode+0x3d/0x360
[1697798.849110] writeback_sb_inodes+0x1f5/0x4d0
[1697798.849113] __writeback_inodes_wb+0x4c/0xf0
[1697798.849116] wb_writeback+0x204/0x2f0
[1697798.849118] wb_workfn+0x2aa/0x4f0
[1697798.849120] ? check_preempt_curr+0x5d/0x70
[1697798.850338] ? ttwu_do_wakeup+0x17/0x170
[1697798.850341] process_one_work+0x1c7/0x380
[1697798.851206] worker_thread+0x4d/0x380
[1697798.851209] ? rescuer_thread+0x3a0/0x3a0
[1697798.851212] kthread+0xda/0x100
[1697798.851214] ? kthread_complete_and_exit+0x20/0x20
[1697798.851217] ret_from_fork+0x22/0x30
[1697798.851222] </TASK>
[1699725.731343] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[1699725.731948] rcu: 1-...0: (10 ticks this GP)
idle=290c/1/0x4000000000000000 softirq=5774501/5774503 fqs=1038
[1699725.734568] (detected by 0, t=5256 jiffies, g=10250969, q=168
ncpus=2)
[1699725.734574] Sending NMI from CPU 0 to CPUs 1:
[1699727.360196] NMI backtrace for cpu 1
[1699727.360204] CPU: 1 PID: 125321 Comm: systemd-journal Tainted: G
L 6.1.0-28-amd64 #1 Debian 6.1.119-1
[1699727.360210] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
[1699727.364642] RIP: 0010:asm_sysvec_reschedule_ipi+0x0/0x20
[1699727.364653] Code: e9 15 07 00 00 0f 1f 44 00 00 0f 01 ca fc 6a ff e8 d5 05
00 00 48 89 c4 48 89 e7 e8 fa f9 e2 ff e9 f5 06 00 00 0f 1f 44 00 00 <0f> 01 ca
fc 6a ff e8 b5 05 00 00 48 89 c4 48 89 e7 e8 fa 1b e3 ff
[1699727.364655] RSP: 0000:fffffe000003dfd8 EFLAGS: 00000006
[1699727.364658] RAX: 0000000000000001 RBX: 0000000002391ee8 RCX:
0000000002391ee8
[1699727.364659] RDX: 0000000000000001 RSI: 00007ff4a6d91ee8 RDI:
00005586b9e9d2d0
[1699727.364660] RBP: 00005586b9e9d2d0 R08: 00005586b9eaa2f0 R09:
00005586b9e9d308
[1699727.364661] R10: 0000000000000003 R11: 00007fffd813ad60 R12:
00007fffd813a7a0
[1699727.364661] R13: 00007fffd813a700 R14: 0000000000000001 R15:
00007ff4a6d91ee8
[1699727.364665] FS: 00007ff4b513c5c0(0000) GS:ffff920fb9d00000(0000)
knlGS:ffff920fb9d00000
[1699727.364666] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1699727.364667] CR2: 00007ff4ac2de950 CR3: 00000001053f6000 CR4:
00000000003506e0
[1699727.364671] Call Trace:
