Saw this on U18 based nodes in gateway roles running
4.15.0-{211,212}-generic. It seems to be fixed in 4.15.0-213-generic.-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed in Ubuntu. https://bugs.launchpad.net/bugs/2018960 Title: linux-image-5.4.0-149-generic (regression): 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 Status in linux-signed package in Ubuntu: Fix Released Status in linux-signed-kvm package in Ubuntu: Fix Released Bug description: After upgrading and rebooting this Ubuntu 20.04 LTS server (Ubuntu Focal), I noticed that it was suddenly getting a bunch of kernel log (dmesg) reports like: WARNING: CPU: 4 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 while investigating I determined that it is currently running the focal-proposed kernel (linux-image-5.4.0-149-generic), which it turns out was enabled for this server (clearly it seemed like a good idea at the time). I'm not expecting focal-proposed to be fixed as if it were a release package, but since I couldn't find any reports on Launchpad I figured I should let y'all know this focal-proposed package could do with some additional work before it's actually released :-) There have been at least 80 such reports in the last 5 hours since the server was rebooted, differing only by the CPU core and the process reported, although it seems the last one was a couple of hours ago, so I guess it's traffic dependent/timing dependent. ewen@naosr620:~$ uptime 16:27:32 up 5:19, 1 user, load average: 0.08, 0.14, 0.06 ewen@naosr620:~$ dmesg -t | grep WARNING | sed 's/CPU: [0-9]*/CPU: N/; s/PID: [0-9]*/PID: N/;' | uniq -c 88 WARNING: CPU: N PID: N at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 ewen@naosr620:~$ Ubuntu Release: ewen@naosr620:~$ lsb_release -rd Description: Ubuntu 20.04.6 LTS Release: 20.04 ewen@naosr620:~$ Kernel/package version affected: ewen@naosr620:~$ uname -a Linux naosr620 5.4.0-149-generic #166-Ubuntu SMP Tue Apr 18 16:51:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ewen@naosr620:~$ dpkg -l | grep linux-image | grep 149 ii linux-image-5.4.0-149-generic 5.4.0-149.166 amd64 Signed kernel image generic ii linux-image-generic 5.4.0.149.147 amd64 Generic Linux kernel image ewen@naosr620:~$ apt-cache policy linux-image-5.4.0-149-generic linux-image-5.4.0-149-generic: Installed: 5.4.0-149.166 Candidate: 5.4.0-149.166 Version table: *** 5.4.0-149.166 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status ewen@naosr620:~$ apt-cache policy linux-image-generic linux-image-generic: Installed: 5.4.0.149.147 Candidate: 5.4.0.149.147 Version table: *** 5.4.0.149.147 500 500 https://mirror.fsmg.org.nz/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 5.4.0.148.146 500 500 https://mirror.fsmg.org.nz/ubuntu focal-updates/main amd64 Packages 500 https://mirror.fsmg.org.nz/ubuntu focal-security/main amd64 Packages 5.4.0.26.32 500 500 https://mirror.fsmg.org.nz/ubuntu focal/main amd64 Packages ewen@naosr620:~$ ewen@naosr620:~$ apt-cache show linux-image-5.4.0-149-generic | grep Source: Source: linux-signed ewen@naosr620:~$ Full example dmesg, including stack trace (they all seem to be WARNINGs, and other than filling dmesg / system logs the system "appears to be running okay", so I'm not going to rush another reboot now -- near end of business day): ewen@naosr620:~$ date Tue 09 May 2023 16:34:56 NZST ewen@naosr620:~$ dmesg -T | tail -100 | grep -B 150 "end trace" | grep -A 999 "cut here" [Tue May 9 14:21:18 2023] ------------[ cut here ]------------ [Tue May 9 14:21:18 2023] WARNING: CPU: 10 PID: 0 at net/core/stream.c:212 sk_stream_kill_queues+0xcf/0xe0 [Tue May 9 14:21:18 2023] Modules linked in: mpt3sas raid_class scsi_transport_sas mptctl mptbase vhost_net vhost tap ip6t_REJECT nf_reject_ipv6 ip6table_mangle ip6table_nat ip6table_raw nf_log_ipv6 xt_recent ipt_REJECT nf_reject_ipv4 xt_hashlimit xt_addrtype xt_multiport xt_comment xt_conntrack xt_mark iptable_mangle xt_MASQUERADE iptable_nat xt_CT xt_tcpudp iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc ebtable_filter nf_nat_h323 ebtables nf_nat_ftp nf_nat_amanda ts_kmp ip6table_filter nf_conntrack_amanda nf_nat ip6_tables nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nfnetlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bpfilter dell_rbu nls_iso8859_1 ipmi_ssif input_leds joydev cdc_ether usbnet mii cdc_acm intel_rapl_ms r intel_rapl_common [Tue May 9 14:21:18 2023] sb_edac x86_pkg_temp_thermal intel_powerclamp binfmt_misc coretemp dcdbas kvm_intel kvm rapl intel_cstate mei_me mei ipmi_si ipmi_devintf mac_hid ipmi_msghandler acpi_power_meter 8021q garp mrp bridge stp llc sch_fq_codel ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid mgag200 drm_vram_helper ttm crct10dif_pclmul crc32_pclmul drm_kms_helper ghash_clmulni_intel syscopyarea aesni_intel sysfillrect sysimgblt ixgbe fb_sys_fops crypto_simd xfrm_algo cryptd mdio drm glue_helper igb megaraid_sas dca i2c_algo_bit ahci libahci lpc_ich wmi [Tue May 9 14:21:18 2023] CPU: 10 PID: 0 Comm: swapper/10 Tainted: G W 5.4.0-149-generic #166-Ubuntu [Tue May 9 14:21:18 2023] Hardware name: Dell Inc. PowerEdge R620/01W23F, BIOS 2.7.0 05/23/2018 [Tue May 9 14:21:18 2023] RIP: 0010:sk_stream_kill_queues+0xcf/0xe0 [Tue May 9 14:21:18 2023] Code: c0 75 21 85 f6 75 23 5b 41 5c 5d c3 48 89 df e8 87 0f ff ff 8b 83 48 01 00 00 8b b3 00 01 00 00 85 c0 74 df 0f 0b 85 f6 74 dd <0f> 0b 5b 41 5c 5d c3 0f 0b eb a8 66 0f 1f 44 00 00 0f 1f 44 00 00 [Tue May 9 14:21:18 2023] RSP: 0018:ffffb4044665c9a8 EFLAGS: 00010206 [Tue May 9 14:21:18 2023] RAX: 0000000000000000 RBX: ffff8cbef55a5f00 RCX: 000000000002546e [Tue May 9 14:21:18 2023] RDX: ffffffffaba54e30 RSI: 0000000000000700 RDI: ffff8cbef55a5f00 [Tue May 9 14:21:18 2023] RBP: ffffb4044665c9b8 R08: 0000000000000000 R09: 0000000000000000 [Tue May 9 14:21:18 2023] R10: 0000000000030000 R11: 000000000c000000 R12: ffff8cbef55a5fd0 [Tue May 9 14:21:18 2023] R13: 0000000000000000 R14: 0000000000000006 R15: 0000000000000079 [Tue May 9 14:21:18 2023] FS: 0000000000000000(0000) GS:ffff8cbf7f740000(0000) knlGS:0000000000000000 [Tue May 9 14:21:18 2023] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue May 9 14:21:18 2023] CR2: 00000000b5ea06e8 CR3: 00000012be20a002 CR4: 00000000001626e0 [Tue May 9 14:21:18 2023] Call Trace: [Tue May 9 14:21:18 2023] <IRQ> [Tue May 9 14:21:18 2023] inet_csk_destroy_sock+0x64/0x150 [Tue May 9 14:21:18 2023] tcp_done+0xbc/0x120 [Tue May 9 14:21:18 2023] tcp_time_wait+0x1a2/0x2c0 [Tue May 9 14:21:18 2023] tcp_fin+0x14e/0x170 [Tue May 9 14:21:18 2023] tcp_data_queue+0x437/0x690 [Tue May 9 14:21:18 2023] tcp_rcv_state_process+0x267/0x740 [Tue May 9 14:21:18 2023] tcp_v6_do_rcv+0x1c5/0x450 [Tue May 9 14:21:18 2023] tcp_v6_rcv+0xc2b/0xd10 [Tue May 9 14:21:18 2023] ip6_protocol_deliver_rcu+0xd3/0x4e0 [Tue May 9 14:21:18 2023] ip6_input_finish+0x15/0x20 [Tue May 9 14:21:18 2023] ip6_input+0xa2/0xb0 [Tue May 9 14:21:18 2023] ? ip6_protocol_deliver_rcu+0x4e0/0x4e0 [Tue May 9 14:21:18 2023] ip6_sublist_rcv_finish+0x3d/0x50 [Tue May 9 14:21:18 2023] ip6_sublist_rcv+0x1aa/0x250 [Tue May 9 14:21:18 2023] ? ip6_rcv_finish_core.isra.0+0xa0/0xa0 [Tue May 9 14:21:18 2023] ipv6_list_rcv+0x112/0x140 [Tue May 9 14:21:18 2023] __netif_receive_skb_list_core+0x1a4/0x250 [Tue May 9 14:21:18 2023] netif_receive_skb_list_internal+0x1a1/0x2b0 [Tue May 9 14:21:18 2023] gro_normal_list.part.0+0x1e/0x40 [Tue May 9 14:21:18 2023] napi_complete_done+0x91/0x130 [Tue May 9 14:21:18 2023] igb_poll+0x7c/0x350 [igb] [Tue May 9 14:21:18 2023] net_rx_action+0x142/0x390 [Tue May 9 14:21:18 2023] __do_softirq+0xd1/0x2c1 [Tue May 9 14:21:18 2023] irq_exit+0xae/0xb0 [Tue May 9 14:21:18 2023] do_IRQ+0x5a/0xf0 [Tue May 9 14:21:18 2023] common_interrupt+0xf/0xf [Tue May 9 14:21:18 2023] </IRQ> [Tue May 9 14:21:18 2023] RIP: 0010:cpuidle_enter_state+0xc5/0x450 [Tue May 9 14:21:18 2023] Code: ff e8 ff da 83 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 65 03 00 00 31 ff e8 72 f0 89 ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8f 02 00 00 49 63 cd 4c 8b 7d d0 4c 2b 7d c8 48 8d [Tue May 9 14:21:18 2023] RSP: 0018:ffffb4044631be38 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffde [Tue May 9 14:21:18 2023] RAX: ffff8cbf7f76fe80 RBX: ffffffffab95a2e0 RCX: 000000000000001f [Tue May 9 14:21:18 2023] RDX: 0000000000000000 RSI: 00000000313b2510 RDI: 0000000000000000 [Tue May 9 14:21:18 2023] RBP: ffffb4044631be78 R08: 00000a8a4770ff97 R09: 0000000000000001 [Tue May 9 14:21:18 2023] R10: ffff8cbf7f76eb80 R11: ffff8cbf7f76eb60 R12: ffffd3f43f940c00 [Tue May 9 14:21:18 2023] R13: 0000000000000004 R14: 0000000000000004 R15: ffffd3f43f940c00 [Tue May 9 14:21:18 2023] ? cpuidle_enter_state+0xa1/0x450 [Tue May 9 14:21:18 2023] cpuidle_enter+0x2e/0x40 [Tue May 9 14:21:18 2023] call_cpuidle+0x23/0x40 [Tue May 9 14:21:18 2023] do_idle+0x1dd/0x270 [Tue May 9 14:21:18 2023] cpu_startup_entry+0x20/0x30 [Tue May 9 14:21:18 2023] start_secondary+0x173/0x1d0 [Tue May 9 14:21:18 2023] secondary_startup_64+0xa4/0xb0 [Tue May 9 14:21:18 2023] ---[ end trace 55353b04455fda06 ]--- ewen@naosr620:~$ (If it gets bad I'll do something about rebooting it onto the latest actually released kernel image; if it's fairly quiet I might leave it until the next focal-proposed kernel package comes out) FTR, this is a fairly production, remote (colo) server so my willingness to try expeimental kernel versions to "see if this is the problem" is low. But if there's more information I can gather from the running kernel that would help, let me know and I'll try to get it for you. Also FTR the server hadn't been rebooted for a while, so I'm pretty sure it was running linux-image-5.4.0-145-generic before the reboot (without these dmesg reports). Ewen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/2018960/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
