** Changed in: linux (Ubuntu) Status: Expired => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987269
Title: [5.18.0 mainline] mlx5 deadlock Status in linux package in Ubuntu: Incomplete Bug description: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy $ uname -a Linux pc1-rb3-n3 5.18.0-051800-generic #202205222030 SMP PREEMPT_DYNAMIC Sun May 22 20:33:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.735461] INFO: task ovn-controller:30439 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.742835] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.749051] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757010] task:ovn-controller state:D stack: 0 pid:30439 ppid: 1 flags:0x00000002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757019] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757024] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757031] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757045] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757050] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757057] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757064] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757069] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757073] rtnetlink_rcv_msg+0x114/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757082] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757087] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757095] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757099] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757104] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757109] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757117] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757123] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757130] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757137] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757143] ? ___sys_recvmsg+0x99/0x110 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757148] ? netdev_name_node_lookup_rcu+0x6b/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757155] ? fput+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757161] ? __sys_sendmsg+0x98/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757166] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757174] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757180] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757186] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757191] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757194] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757200] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757203] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757210] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757214] ? __x64_sys_write+0x19/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757220] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757224] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757227] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757232] RIP: 0033:0x7f571f927b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757239] RSP: 002b:00007ffd50fa8ec0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757246] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f571f927b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757252] RDX: 0000000000000000 RSI: 00007ffd50fa8f50 RDI: 0000000000000020 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757255] RBP: 00007ffd50fa9c40 R08: 0000000000000000 R09: 000055e1a6a66f60 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757260] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757264] R13: 000055e1a53bc510 R14: 0000000000074f30 R15: 00007ffd50fa8f50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757271] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.757646] INFO: task ovs-vswitchd:45997 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.764817] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.771036] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.778998] task:ovs-vswitchd state:D stack: 0 pid:45997 ppid: 1 flags:0x00000002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779005] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779007] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779009] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779017] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779023] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779029] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779036] ? netlink_unicast+0x305/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779041] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779046] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779050] rtnetlink_rcv_msg+0x114/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779056] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779060] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779065] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779069] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779077] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779083] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779090] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779096] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779101] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779107] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779112] ? _copy_from_user+0x3e/0x60 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779117] ? iovec_from_user.part.0+0x59/0x190 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779121] ? __import_iovec+0x47/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779126] ? ___sys_recvmsg+0x99/0x110 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779131] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779136] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779143] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779149] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779154] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779159] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779163] ? __x64_sys_recvmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779168] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779172] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779177] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779181] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779185] ? asm_sysvec_apic_timer_interrupt+0xa/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779192] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779197] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779201] RSP: 002b:00007ffcefe1f410 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779208] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779213] RDX: 0000000000000000 RSI: 00007ffcefe1f4a0 RDI: 0000000000000141 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779217] RBP: 00007ffcefe201d8 R08: 0000000000000000 R09: 0000000000000020 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779220] R10: 0000000000000001 R11: 0000000000000293 R12: 00007f4014006030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779223] R13: 0000000000000001 R14: 0000000000000003 R15: 00007ffcefe1f4a0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779229] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.779247] INFO: task handler98:46136 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.786162] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.792382] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800327] task:handler98 state:D stack: 0 pid:46136 ppid: 1 flags:0x00004002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800331] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800333] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800334] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800340] ? mlx5dr_send_postsend_action+0x70/0x90 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800429] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800433] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800436] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800441] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800444] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800448] mlx5_tc_ct_flow_offload+0x36/0x60 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800526] mlx5e_tc_rule_offload+0xa3/0x110 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800600] mlx5e_tc_offload_fdb_rules+0x36/0xb0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800668] mlx5e_tc_add_fdb_flow+0x3a2/0x410 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800740] __mlx5e_add_fdb_flow+0x2be/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800806] mlx5e_add_fdb_flow+0x39/0x220 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800871] mlx5e_configure_flower+0x3af/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.800936] mlx5e_rep_setup_tc_cls_flower+0x3a/0x70 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801001] mlx5e_rep_setup_tc_cb+0x58/0x80 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801065] tc_setup_cb_add+0xcf/0x200 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801071] fl_hw_replace_filter+0x15b/0x240 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801077] fl_change+0x8b1/0xbf0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801082] tc_new_tfilter+0x3e5/0xb70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801086] ? skb_queue_tail+0x48/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801091] ? tcf_exts_terse_dump+0xc0/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801094] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801097] ? skb_free_head+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801101] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801103] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801107] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801109] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801112] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801116] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801119] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801122] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801125] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801129] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801132] ? ___sys_sendmsg+0x8e/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801135] ? __import_iovec+0x47/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801138] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801141] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801144] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801148] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801151] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801154] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801157] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801160] ? __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801163] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801166] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801169] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801171] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801174] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801176] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801178] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801179] RSP: 002b:00007f3fcffb2490 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801182] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801183] RDX: 0000000000000000 RSI: 00007f3fcffb2520 RDI: 000000000000013e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801185] RBP: 00007f3fcffb3258 R08: 0000000000000000 R09: 0000000000000134 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801186] R10: 0000000000007463 R11: 0000000000000293 R12: 00007f3ed804cda0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801188] R13: 0000000000000001 R14: 000000000000000f R15: 00007f3fcffb2520 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801191] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.801193] INFO: task handler105:46143 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.808173] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.814381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822319] task:handler105 state:D stack: 0 pid:46143 ppid: 1 flags:0x00004002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822324] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822326] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822328] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822332] ? mlx5dr_send_postsend_action+0x70/0x90 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822408] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822412] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822415] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822419] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822423] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822426] mlx5_tc_ct_flow_offload+0x36/0x60 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822500] mlx5e_tc_rule_offload+0xa3/0x110 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822569] mlx5e_tc_offload_fdb_rules+0x36/0xb0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822635] mlx5e_tc_add_fdb_flow+0x3a2/0x410 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822702] __mlx5e_add_fdb_flow+0x2be/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822771] mlx5e_add_fdb_flow+0x15c/0x220 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822834] mlx5e_configure_flower+0x3af/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822898] mlx5e_rep_setup_tc_cls_flower+0x3a/0x70 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.822989] mlx5e_rep_setup_tc_cb+0x58/0x80 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823054] tc_setup_cb_add+0xcf/0x200 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823058] fl_hw_replace_filter+0x15b/0x240 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823064] fl_change+0x8b1/0xbf0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823070] tc_new_tfilter+0x3e5/0xb70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823077] ? tcf_exts_terse_dump+0xc0/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823080] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823084] ? skb_free_head+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823089] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823094] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823098] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823102] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823106] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823112] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823116] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823119] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823122] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823127] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823131] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823135] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823139] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823143] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823148] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823152] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823155] ? pebs_update_state+0x40/0x2e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823162] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823166] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823170] ? __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823174] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823178] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823182] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823186] RSP: 002b:00007f3faffb2490 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823191] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823195] RDX: 0000000000000000 RSI: 00007f3faffb2520 RDI: 0000000000000144 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823198] RBP: 00007f3faffb3258 R08: 0000000000000000 R09: 0000000000000134 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823201] R10: 0000000000007463 R11: 0000000000000293 R12: 00007f3fbc015dd0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823204] R13: 0000000000000001 R14: 0000000000000003 R15: 00007f3faffb2520 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823208] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.823213] INFO: task revalidator130:46169 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.830553] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.836758] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844692] task:revalidator130 state:D stack: 0 pid:46169 ppid: 1 flags:0x00004002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844696] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844697] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844699] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844702] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844705] schedule_timeout+0x115/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844707] ? __free_pages+0xa1/0xf0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844713] __wait_for_common+0x92/0x180 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844715] ? usleep_range_state+0x90/0x90 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844717] wait_for_completion+0x24/0x30 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844719] flush_workqueue+0x133/0x3e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844725] mlx5_tc_ct_del_ft_cb.part.0+0x120/0x180 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844774] mlx5_tc_ct_delete_flow+0xb7/0xd0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844819] mlx5e_tc_rule_unoffload+0x8f/0xf0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844865] mlx5e_tc_unoffload_fdb_rules+0x36/0x60 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844909] mlx5e_tc_del_fdb_flow+0x227/0x3e0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844952] __mlx5e_tc_del_fdb_peer_flow+0xcc/0xf0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.844994] mlx5e_flow_put+0x63/0xc0 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845035] mlx5e_delete_flower+0x202/0x270 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845077] mlx5e_rep_setup_tc_cls_flower+0x6a/0x70 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845119] mlx5e_rep_setup_tc_cb+0x58/0x80 [mlx5_core] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845160] tc_setup_cb_destroy+0xb5/0x190 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845162] fl_hw_destroy_filter+0x95/0xc0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845166] __fl_delete+0x1d0/0x1f0 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845169] fl_delete+0x39/0x80 [cls_flower] Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845172] tc_del_tfilter+0x385/0x910 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845176] ? tc_setup_cb_add+0x200/0x200 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845178] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845181] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845183] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845185] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845187] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845189] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845191] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845193] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845195] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845197] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845199] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845201] ? __import_iovec+0x47/0x150 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845203] ? ___sys_recvmsg+0x99/0x110 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845205] ? move_addr_to_user+0x4d/0xe0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845207] ? __sys_getsockname+0xcb/0xe0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845209] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845211] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845213] ? pebs_update_state+0x40/0x2e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845216] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845218] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845220] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845222] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845224] ? exit_to_user_mode_prepare+0x37/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845226] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845228] ? __x64_sys_recvmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845230] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845231] ? syscall_exit_to_user_mode+0x26/0x50 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845233] ? __x64_sys_socket+0x17/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845235] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845236] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845237] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845239] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845240] ? sysvec_apic_timer_interrupt+0x4e/0x90 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845242] ? asm_sysvec_apic_timer_interrupt+0xa/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845243] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845245] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845246] RSP: 002b:00007f3f4dff05a0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845247] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845249] RDX: 0000000000000000 RSI: 00007f3f4dff0630 RDI: 0000000000000140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845250] RBP: 00007f3f4dff1368 R08: 0000000000000000 R09: 0000000000000010 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845251] R10: 0000000000000001 R11: 0000000000000293 R12: 00007f3f30019bd0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845251] R13: 0000000000000001 R14: 0000000000000002 R15: 00007f3f4dff0630 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845254] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.845536] INFO: task revalidator133:46172 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.852972] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.859174] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867104] task:revalidator133 state:D stack: 0 pid:46172 ppid: 1 flags:0x00000002 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867109] Call Trace: Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867111] <TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867113] __schedule+0x246/0x5c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867120] schedule+0x55/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867124] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867129] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867133] ? __nla_validate_parse+0x14c/0x1a0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867140] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867143] mutex_lock+0x35/0x40 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867145] rtnl_lock+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867148] tc_get_tfilter+0x11a/0x5d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867151] ? _copy_to_iter+0xcc/0x6c0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867154] ? tc_new_tfilter+0xb70/0xb70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867158] rtnetlink_rcv_msg+0x2fd/0x3f0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867161] ? rtnl_calcit.isra.0+0x140/0x140 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867163] netlink_rcv_skb+0x56/0x100 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867166] rtnetlink_rcv+0x15/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867167] netlink_unicast+0x236/0x350 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867169] netlink_sendmsg+0x25e/0x4d0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867172] sock_sendmsg+0x66/0x70 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867174] ____sys_sendmsg+0x24f/0x290 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867176] ? import_iovec+0x1b/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867178] ? sendmsg_copy_msghdr+0x7c/0xa0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867181] ? iovec_from_user.part.0+0x59/0x190 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867183] ___sys_sendmsg+0x81/0xc0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867185] ? fput+0x13/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867188] ? __sys_recvmsg+0x95/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867191] ? 0xffffffffb5000000 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867193] ? __fget_light+0xa7/0x130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867198] __sys_sendmsg+0x62/0xb0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867201] ? pebs_update_state+0x40/0x2e0 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867204] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867207] __x64_sys_sendmsg+0x1d/0x20 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867210] do_syscall_64+0x5c/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867213] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867216] ? do_syscall_64+0x69/0x80 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867218] entry_SYSCALL_64_after_hwframe+0x44/0xae Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867221] RIP: 0033:0x7f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867224] RSP: 002b:00007f3f2fff44b0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867228] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4187327b4d Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867231] RDX: 0000000000000000 RSI: 00007f3f2fff4540 RDI: 000000000000013f Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867234] RBP: 00007f3f2fff5278 R08: 0000000000000000 R09: 0000000000000024 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867238] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f3f2004b130 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867240] R13: 0000000000000001 R14: 0000000000000005 R15: 00007f3f2fff4540 Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867244] </TASK> Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.867385] INFO: task kworker/u256:0:1812497 blocked for more than 120 seconds. Aug 22 01:15:02 pc1-rb3-n3 kernel: [200712.874888] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.881104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889038] task:kworker/u256:0 state:D stack: 0 pid:1812497 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889043] Workqueue: mlx5e mlx5e_rep_neigh_update [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889103] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889105] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889106] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889108] ? sched_clock_local+0x13/0x90 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889113] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889116] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889118] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889121] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889124] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889126] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889129] mlx5e_rep_neigh_update+0x35/0x1a0 [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889175] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889178] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889181] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889183] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889187] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889190] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889195] </TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.889200] INFO: task kworker/56:0:2429317 blocked for more than 120 seconds. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.896544] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.902740] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910672] task:kworker/56:0 state:D stack: 0 pid:2429317 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910677] Workqueue: ipv6_addrconf addrconf_verify_work Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910684] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910685] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910686] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910689] ? newidle_balance+0x30d/0x480 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910694] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910695] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910698] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910701] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910703] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910705] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910707] addrconf_verify_work+0x12/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910711] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910715] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910718] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910722] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910725] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910728] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910732] </TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.910740] INFO: task kworker/u256:51:2462708 blocked for more than 121 seconds. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.918334] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.924539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932489] task:kworker/u256:51 state:D stack: 0 pid:2462708 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932493] Workqueue: mlx5_fc mlx5e_rep_neigh_stats_work [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932552] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932553] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932555] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932558] ? default_send_IPI_single_phys+0x4c/0x80 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932563] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932565] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932568] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932572] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932575] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932577] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932580] mlx5e_rep_neigh_stats_work+0x20/0x120 [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932632] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932635] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932638] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932640] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932642] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932645] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932648] </TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.932654] INFO: task kworker/u256:29:2732306 blocked for more than 121 seconds. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.940241] Not tainted 5.18.0-051800-generic #202205222030 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.946494] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954437] task:kworker/u256:29 state:D stack: 0 pid:2732306 ppid: 2 flags:0x00004000 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954441] Workqueue: mlx5_fc mlx5e_rep_neigh_stats_work [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954486] Call Trace: Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954487] <TASK> Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954488] __schedule+0x246/0x5c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954491] ? default_send_IPI_single_phys+0x4c/0x80 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954494] schedule+0x55/0xc0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954495] schedule_preempt_disabled+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954498] __mutex_lock.constprop.0+0x2f3/0x4c0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954500] __mutex_lock_slowpath+0x13/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954504] mutex_lock+0x35/0x40 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954507] rtnl_lock+0x15/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954511] mlx5e_rep_neigh_stats_work+0x20/0x120 [mlx5_core] Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954554] process_one_work+0x21f/0x3f0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954557] worker_thread+0x50/0x3e0 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954560] ? rescuer_thread+0x390/0x390 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954564] kthread+0xee/0x120 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954567] ? kthread_complete_and_exit+0x20/0x20 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954570] ret_from_fork+0x22/0x30 Aug 22 01:15:03 pc1-rb3-n3 kernel: [200712.954574] </TASK> To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1987269/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp