On Wed, May 21, 2025 at 5:43 AM Xiongfeng Wang <wangxiongfe...@huawei.com> wrote: > > Hi RCU experts, > > When I ran syskaller in Linux 6.6 with CONFIG_PREEMPT_RCU enabled, I got > the following soft lockup. The Calltrace is too long. I put it in the end. > The issue can also be reproduced in the latest kernel. > > The issue is as follows. CPU3 is waiting for a spin_lock, which is got by > CPU1. > But CPU1 stuck in the following dead loop. > > irq_exit() > __irq_exit_rcu() > /* in_hardirq() returns false after this */ > preempt_count_sub(HARDIRQ_OFFSET) > tick_irq_exit() > tick_nohz_irq_exit() > tick_nohz_stop_sched_tick() > trace_tick_stop() /* a bpf prog is hooked on this trace point > */ > __bpf_trace_tick_stop() > bpf_trace_run2() > rcu_read_unlock_special() > /* will send a IPI to itself */ > irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu); > > /* after interrupt is enabled again, the irq_work is called */ > asm_sysvec_irq_work() > sysvec_irq_work() > irq_exit() /* after handled the irq_work, we again enter into irq_exit() */ > __irq_exit_rcu() > ...skip... > /* we queue a irq_work again, and enter a dead loop */ > irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
This seems legitimate, Boqun and I were just talking about it. He may share more thoughts but here are a few: Maybe we can delay subsequent clearing of the flag in rcu_preempt_deferred_qs_handler() using a timer and an exponential back-off? That way we are not sending too many self-IPIs. And reset the process at the end of a grace period. Or just don't send subsequent self-IPIs if we just sent one for the rdp. Chances are, if we did not get the scheduler's attention during the first one, we may not in subsequent ones I think. Plus we do send other IPIs already if the grace period was over extended (from the FQS loop), maybe we can tweak that? thanks, - Joel > > I am not familiar with RCU code and don't know how to solve this problem. > Any guidance would be appreciated. > > Thanks, > Xiongfeng > > > [ 1901.634222][ C3] watchdog: BUG: soft lockup - CPU#3 stuck for 913s! > [syz.8.781:8076] > [ 1901.634947][ C3] Modules linked in: > [ 1901.635261][ C3] CPU: 3 PID: 8076 Comm: syz.8.781 Tainted: G > L > 6.6.0+ #1873 > [ 1901.635981][ C3] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, > 1996), BIOS 0.0.0 02/06/2015 > [ 1901.636735][ C3] RIP: 0010:queued_spin_lock_slowpath+0x109/0x9c0 > [ 1901.637281][ C3] Code: 00 48 8b 44 24 60 65 48 2b 04 25 28 00 00 00 0f > 85 > ca 07 00 00 48 83 c4 68 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc f3 90 > <eb> > 85 81 fe 00 01 00 00 0f 84 dc 00 00 00 81 fe ff 00 00 00 0f 87 > [ 1901.638858][ C3] RSP: 0000:ff1100002900f8b0 EFLAGS: 00000202 > [ 1901.639364][ C3] RAX: 0000000000000000 RBX: 0000000000000001 RCX: > ffffffff961f5ecd > [ 1901.640001][ C3] RDX: fffa7c000005f936 RSI: 0000000000000004 RDI: > ffd40000002fc9a8 > [ 1901.640624][ C3] RBP: 0000000000000003 R08: 0000000000000000 R09: > fffa7c000005f935 > [ 1901.641255][ C3] R10: ffd40000002fc9ab R11: 0000000000000000 R12: > fffa7c000005f935 > [ 1901.641905][ C3] R13: 0000000000000001 R14: 1fe2200005201f17 R15: > ffd40000002fc9a8 > [ 1901.642527][ C3] FS: 000000002316c500(0000) GS:ff11000107980000(0000) > knlGS:0000000000000000 > [ 1901.643272][ C3] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 1901.643818][ C3] CR2: 0000000020000000 CR3: 0000000036592004 CR4: > 0000000000771ee0 > [ 1901.644469][ C3] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 1901.645105][ C3] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 1901.645724][ C3] PKRU: 80000000 > [ 1901.646010][ C3] Call Trace: > [ 1901.646287][ C3] <TASK> > [ 1901.646526][ C3] ? __pfx_queued_spin_lock_slowpath+0x10/0x10 > [ 1901.647053][ C3] _raw_spin_lock+0xd0/0xe0 > [ 1901.647450][ C3] ? __pfx__raw_spin_lock+0x10/0x10 > [ 1901.647883][ C3] __pte_offset_map_lock+0x129/0x270 > [ 1901.648335][ C3] ? __pfx___pte_offset_map_lock+0x10/0x10 > [ 1901.648820][ C3] migration_entry_wait+0x85/0x270 > [ 1901.649247][ C3] ? __pfx_migration_entry_wait+0x10/0x10 > [ 1901.649705][ C3] ? __pfx___schedule+0x10/0x10 > [ 1901.650122][ C3] ? __hrtimer_start_range_ns+0x355/0xd00 > [ 1901.650623][ C3] do_swap_page+0x36d2/0x4670 > [ 1901.651016][ C3] ? plist_del+0x24c/0x450 > [ 1901.651453][ C3] ? __pte_offset_map+0x2b9/0x3d0 > [ 1901.651981][ C3] ? __pfx_do_swap_page+0x10/0x10 > [ 1901.652432][ C3] ? __pfx_default_wake_function+0x10/0x10 > [ 1901.652947][ C3] handle_pte_fault+0x5ae/0x7b0 > [ 1901.653387][ C3] __handle_mm_fault+0xf9e/0x17f0 > [ 1901.653839][ C3] ? __pfx___handle_mm_fault+0x10/0x10 > [ 1901.654335][ C3] ? lock_vma_under_rcu+0x377/0x4d0 > [ 1901.654797][ C3] ? __pfx_lock_vma_under_rcu+0x10/0x10 > [ 1901.655294][ C3] handle_mm_fault+0x4bf/0xa20 > [ 1901.655725][ C3] exc_page_fault+0x20b/0x7f0 > [ 1901.656150][ C3] asm_exc_page_fault+0x26/0x30 > [ 1901.656582][ C3] RIP: 0033:0x51d788 > [ 1901.656927][ C3] Code: 66 89 74 17 02 88 0f c3 c5 fa 6f 06 c5 fa 6f 4c > 16 > f0 c5 fa 7f 07 c5 fa 7f 4c 17 f0 c3 0f 1f 44 00 00 48 8b 4c 16 f8 48 8b 36 > <48> > 89 37 48 89 4c 17 f8 c3 62 e1 fe 28 6f 54 16 ff 62 e1 fe 28 6f > [ 1901.658618][ C3] RSP: 002b:00007fff991f9118 EFLAGS: 00010202 > [ 1901.659157][ C3] RAX: 0000000020000000 RBX: 0000000000000004 RCX: > 002367732f766564 > [ 1901.659856][ C3] RDX: 0000000000000009 RSI: 2367732f7665642f RDI: > 0000000020000000 > [ 1901.660553][ C3] RBP: 0000000000797ba0 R08: 00007f10a7800000 R09: > 0000000000000001 > [ 1901.661240][ C3] R10: 0000000000000001 R11: 0000000000000009 R12: > 0000000000000032 > [ 1901.661906][ C3] R13: 00000000000e2c77 R14: 0000000000795fa0 R15: > 0000000000795fac > [ 1901.662589][ C3] </TASK> > [ 1901.662864][ C3] Sending NMI from CPU 3 to CPUs 0-2: > [ 1901.663358][ C1] NMI backtrace for cpu 1 > [ 1901.663365][ C1] AAA2 1901441189248 1901441163169 0 0 0 1901441162197 > [ 1901.663374][ C1] AAA2 1901441197984 993372484411 993372484804 > 993372484875 > 928887973527 928887973577 928887974425 // handle_softirqs > [ 1901.663383][ C1] softirq_handle_start: > [ 1901.663385][ C1] 0 > [ 1901.663387][ C1] 928330006990 > [ 1901.663390][ C1] 924221567459 > [ 1901.663392][ C1] 928817263100 > [ 1901.663395][ C1] 928328744939 > [ 1901.663397][ C1] 0 > [ 1901.663399][ C1] 921062008817 > [ 1901.663401][ C1] 928879010614 > [ 1901.663403][ C1] 0 > [ 1901.663405][ C1] 928887973635 > [ 1901.663407][ C1] > [ 1901.663409][ C1] softirq_handle_end_time: > [ 1901.663410][ C1] 0 > [ 1901.663412][ C1] 928330009295 > [ 1901.663415][ C1] 924221568491 > [ 1901.663417][ C1] 928817511439 > [ 1901.663419][ C1] 928328806233 > [ 1901.663421][ C1] 0 > [ 1901.663423][ C1] 921062009968 > [ 1901.663425][ C1] 928879011708 > [ 1901.663427][ C1] 0 > [ 1901.663429][ C1] 928887974243 > [ 1901.663432][ C1] > [ 1901.663434][ C1] CPU: 1 PID: 8079 Comm: syz.8.781 Tainted: G > L > 6.6.0+ #1873 > [ 1901.663444][ C1] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, > 1996), BIOS 0.0.0 02/06/2015 > [ 1901.663448][ C1] RIP: 0010:irq_work_claim+0x5c/0xa0 > [ 1901.663464][ C1] Code: 14 02 48 89 e8 83 e0 07 83 c0 03 38 d0 7c 04 84 > d2 > 75 44 8b 5b 08 e8 c3 74 e7 ff 89 da 41 89 dc 89 d8 83 ca 23 f0 0f b1 55 00 > <41> > 0f 94 c5 31 ff 89 c3 44 89 ee e8 74 6a e7 ff 45 84 ed 74 d7 e8 > [ 1901.663473][ C1] RSP: 0018:ff11000107889c68 EFLAGS: 00000046 > [ 1901.663480][ C1] RAX: 0000000000000000 RBX: 0000000000000000 RCX: > ffffffff8dc11a4d > [ 1901.663486][ C1] RDX: 0000000000000023 RSI: 0000000000000004 RDI: > ff110001078c6838 > [ 1901.663491][ C1] RBP: ff110001078c6838 R08: 0000000000000001 R09: > ffe21c0020f18d07 > [ 1901.663496][ C1] R10: ff110001078c683b R11: 0000000000000000 R12: > 0000000000000000 > [ 1901.663502][ C1] R13: ff110001078c6bf0 R14: 0000000000000000 R15: > ffffffff99f43000 > [ 1901.663507][ C1] FS: 00007f10a79556c0(0000) GS:ff11000107880000(0000) > knlGS:0000000000000000 > [ 1901.663517][ C1] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 1901.663523][ C1] CR2: 0000000000638300 CR3: 0000000036592002 CR4: > 0000000000771ee0 > [ 1901.663529][ C1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > [ 1901.663534][ C1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: > 0000000000000400 > [ 1901.663539][ C1] PKRU: 00000000 > [ 1901.663542][ C1] Call Trace: > [ 1901.663546][ C1] <IRQ> > [ 1901.663551][ C1] irq_work_queue_on+0x59/0x130 > [ 1901.663563][ C1] rcu_read_unlock_special+0x46a/0x530 > [ 1901.663577][ C1] bpf_trace_run2+0xf7/0x220 > [ 1901.663588][ C1] ? __pfx_bpf_trace_run2+0x10/0x10 > [ 1901.663599][ C1] ? __pfx_ep_poll_callback+0x10/0x10 > [ 1901.663613][ C1] __bpf_trace_tick_stop+0xb4/0xf0 > [ 1901.663623][ C1] ? __pfx___bpf_trace_tick_stop+0x10/0x10 > [ 1901.663633][ C1] ? __pfx_sched_clock_cpu+0x10/0x10 > [ 1901.663643][ C1] ? __wake_up+0x44/0x60 > [ 1901.663657][ C1] check_tick_dependency+0x362/0x670 > [ 1901.663673][ C1] __tick_nohz_full_update_tick+0xd1/0x220 > [ 1901.663683][ C1] tick_nohz_irq_exit+0x22c/0x2a0 > [ 1901.663694][ C1] sysvec_irq_work+0x36/0x80 > [ 1901.663705][ C1] asm_sysvec_irq_work+0x1a/0x20 > [ 1901.663720][ C1] RIP: 0010:handle_softirqs+0x230/0x8d0 > [ 1901.663734][ C1] Code: 24 f5 80 5d 84 98 e8 1f 9b 27 00 4c 89 e2 48 c1 > ea > 03 42 80 3c 3a 00 0f 85 d5 05 00 00 49 89 04 24 fb 65 44 8b 35 0c 2c 93 72 > <4d> > 63 f6 49 c7 c4 20 0c 03 00 4a 8d 3c f5 80 5d 84 98 48 89 f8 48 > [ 1901.663742][ C1] RSP: 0018:ff11000107889f68 EFLAGS: 00000246 > [ 1901.663748][ C1] RAX: 000000e7499d350b RBX: 0000000000000000 RCX: > 0000000000000018 > [ 1901.663754][ C1] RDX: 1fe2200020f16185 RSI: 00000000001e2170 RDI: > 0000000000000004 > [ 1901.663759][ C1] RBP: ff1100003ae422c0 R08: 0000000000000001 R09: > ffe21c0020f16239 > [ 1901.663765][ C1] R10: 00000000001e2170 R11: 3030303030302052 R12: > ff110001078b0c28 > [ 1901.663770][ C1] R13: 0000000000400140 R14: 0000000000000001 R15: > dffffc0000000000 > [ 1901.663779][ C1] ? handle_softirqs+0x211/0x8d0 > [ 1901.663792][ C1] ? update_sibling_normalize_runtime+0x1a/0x410 > [ 1901.663807][ C1] irq_exit_rcu+0x134/0x190 > [ 1901.663820][ C1] sysvec_irq_work+0x6a/0x80 > [ 1901.663830][ C1] </IRQ> > [ 1901.663833][ C1] <TASK> > [ 1901.663835][ C1] asm_sysvec_irq_work+0x1a/0x20 > [ 1901.663849][ C1] RIP: 0010:rcu_read_unlock_special+0x186/0x530 > [ 1901.663861][ C1] Code: 0f 85 01 01 00 00 4d 85 f6 0f 84 6a 01 00 00 40 > 84 > ed 0f 84 61 01 00 00 bf 09 00 00 00 e8 c2 45 df ff fb 65 8b 2d 26 65 72 72 > <48> > 63 ed 48 c7 c3 40 1a 03 00 48 b8 00 00 00 00 00 fc ff df 48 8d > [ 1901.663869][ C1] RSP: 0018:ff110000380b6fe8 EFLAGS: 00000283 > [ 1901.663875][ C1] RAX: 0000000000000001 RBX: ff110001078c6800 RCX: > ffffffff8d7c6a90 > [ 1901.663881][ C1] RDX: 0000000000000001 RSI: 0000000000000046 RDI: > ff11000100e33084 > [ 1901.663886][ C1] RBP: 0000000000000001 R08: 0000000000000000 R09: > fffffbfff3716ab4 > [ 1901.663891][ C1] R10: 0000000000000000 R11: 0000000000000001 R12: > 0000000000000000 > [ 1901.663896][ C1] R13: dffffc0000000000 R14: 0000000000000200 R15: > ffffffff99f43000 > [ 1901.663903][ C1] ? ttwu_do_activate+0x520/0x6f0 > [ 1901.663918][ C1] ? rcu_read_unlock_special+0x17e/0x530 > [ 1901.663930][ C1] page_vma_mapped_walk+0x1e00/0x2b50 > [ 1901.663948][ C1] ? __netlink_deliver_tap_skb+0x352/0x4d0 > [ 1901.663959][ C1] ? __pfx_page_vma_mapped_walk+0x10/0x10 > [ 1901.663974][ C1] ? netlink_ack+0x614/0xba0 > [ 1901.663986][ C1] try_to_migrate_one+0x3f0/0x2cb0 > [ 1901.663999][ C1] ? __sys_sendmsg+0xee/0x1b0 > [ 1901.664015][ C1] ? __pfx_try_to_migrate_one+0x10/0x10 > [ 1901.664028][ C1] ? __orc_find+0x109/0x140 > [ 1901.664041][ C1] ? arch_stack_walk+0x92/0x160 > [ 1901.664058][ C1] ? __anon_vma_interval_tree_subtree_search+0x171/0x1f0 > [ 1901.664080][ C1] ? __pfx_try_to_migrate_one+0x10/0x10 > [ 1901.664094][ C1] rmap_walk_anon+0x2b0/0x980 > [ 1901.664108][ C1] try_to_migrate+0x19f/0x350 > [ 1901.664122][ C1] ? __pfx_try_to_migrate+0x10/0x10 > [ 1901.664135][ C1] ? __pfx_try_to_migrate_one+0x10/0x10 > [ 1901.664148][ C1] ? __pfx_folio_not_mapped+0x10/0x10 > [ 1901.664159][ C1] ? __pfx_folio_lock_anon_vma_read+0x10/0x10 > [ 1901.664173][ C1] ? __pfx_invalid_migration_vma+0x10/0x10 > [ 1901.664184][ C1] ? folio_total_mapcount+0xb4/0x210 > [ 1901.664197][ C1] split_huge_page_to_list_to_order+0xc84/0x17f0 > [ 1901.664213][ C1] ? __pfx_split_huge_page_to_list_to_order+0x10/0x10 > [ 1901.664226][ C1] ? __orc_find+0x109/0x140 > [ 1901.664240][ C1] ? __orc_find+0x109/0x140 > [ 1901.664253][ C1] ? entry_SYSCALL_64_after_hwframe+0x77/0xe2 > [ 1901.664269][ C1] madvise_cold_or_pageout_pte_range+0x1966/0x2450 > [ 1901.664284][ C1] ? arch_stack_walk+0x92/0x160 > [ 1901.664298][ C1] ? __read_once_word_nocheck+0x9/0x20 > [ 1901.664311][ C1] ? deref_stack_reg+0x1a4/0x2b0 > [ 1901.664326][ C1] ? entry_SYSCALL_64_after_hwframe+0x77/0xe2 > [ 1901.664340][ C1] ? __pfx_madvise_cold_or_pageout_pte_range+0x10/0x10 > [ 1901.664354][ C1] ? __read_once_word_nocheck+0x9/0x20 > [ 1901.664368][ C1] ? __orc_find+0x109/0x140 > [ 1901.664381][ C1] ? __pfx_madvise_cold_or_pageout_pte_range+0x10/0x10 > [ 1901.664395][ C1] walk_pmd_range.isra.0+0x240/0x720 > [ 1901.664405][ C1] ? __read_once_word_nocheck+0x9/0x20 > [ 1901.664419][ C1] walk_pud_range.isra.0+0x3d3/0x6c0 > [ 1901.664431][ C1] walk_p4d_range+0x2ef/0x4f0 > [ 1901.664441][ C1] walk_pgd_range+0x27e/0x530 > [ 1901.664452][ C1] __walk_page_range+0x4ab/0x5a0 > [ 1901.664461][ C1] ? find_vma+0x81/0xb0 > [ 1901.664469][ C1] ? __pfx_find_vma+0x10/0x10 > [ 1901.664477][ C1] ? folios_put_refs+0x510/0x740 > [ 1901.664487][ C1] ? walk_page_test+0xa0/0x190 > [ 1901.664503][ C1] walk_page_range+0x2a0/0x530 > [ 1901.664513][ C1] ? __pfx_walk_page_range+0x10/0x10 > [ 1901.664522][ C1] ? vtime_task_switch_generic+0x278/0x5a0 > [ 1901.664535][ C1] ? folio_batch_move_lru+0x2b8/0x3d0 > [ 1901.664544][ C1] ? __pfx_lru_add_fn+0x10/0x10 > [ 1901.664554][ C1] madvise_pageout_page_range+0x1cc/0x6d0 > [ 1901.664567][ C1] ? __pfx_madvise_pageout_page_range+0x10/0x10 > [ 1901.664582][ C1] madvise_pageout+0x1f4/0x400 > [ 1901.664595][ C1] ? __pfx_madvise_pageout+0x10/0x10 > [ 1901.664608][ C1] ? futex_wait+0x552/0x680 > [ 1901.664624][ C1] ? __sanitizer_cov_trace_switch+0x54/0x90 > [ 1901.664635][ C1] ? __sanitizer_cov_trace_switch+0x54/0x90 > [ 1901.664645][ C1] ? mas_prev_setup.constprop.0+0xb4/0x530 > [ 1901.664661][ C1] madvise_vma_behavior+0x8fa/0xe30 > [ 1901.664675][ C1] ? __pfx_madvise_vma_behavior+0x10/0x10 > [ 1901.664688][ C1] ? find_vma_prev+0xf5/0x170 > [ 1901.664698][ C1] ? __pfx_find_vma_prev+0x10/0x10 > [ 1901.664711][ C1] ? do_futex+0x135/0x360 > [ 1901.664724][ C1] do_madvise+0x3af/0x650 > [ 1901.664737][ C1] ? __pfx_do_madvise+0x10/0x10 > [ 1901.664749][ C1] ? __se_sys_futex+0xf7/0x390 > [ 1901.664762][ C1] ? kvm_steal_clock+0xca/0x100 > [ 1901.664778][ C1] ? get_vtime_delta+0xd7/0x250 > [ 1901.664793][ C1] __x64_sys_madvise+0xaf/0x120 > [ 1901.664805][ C1] ? __ct_user_exit+0x70/0xe0 > [ 1901.664819][ C1] do_syscall_64+0x59/0x110 > [ 1901.664834][ C1] entry_SYSCALL_64_after_hwframe+0x78/0xe2 > [ 1901.664849][ C1] RIP: 0033:0x54d2cd > [ 1901.664856][ C1] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e > fa > 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 > <48> > 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 > [ 1901.664864][ C1] RSP: 002b:00007f10a7955048 EFLAGS: 00000246 ORIG_RAX: > 000000000000001c > [ 1901.664873][ C1] RAX: ffffffffffffffda RBX: 0000000000795fa0 RCX: > 000000000054d2cd > [ 1901.664878][ C1] RDX: 0000000000000015 RSI: 0000000000003000 RDI: > 0000000020001000 > [ 1901.664884][ C1] RBP: 0000000000000000 R08: 0000000000000000 R09: > 0000000000000000 > [ 1901.664889][ C1] R10: 0000000000000000 R11: 0000000000000246 R12: > 0000000000795fac > [ 1901.664894][ C1] R13: 0000000000000000 R14: 0000000000795fa0 R15: > 00007f10a7935000 >
