On Wed, Nov 14, 2018 at 04:03:33AM -0500, Kyungtae Kim wrote:
> We report two crashes in v4.19-rc8 (4.20-rc1 as well, I guess):
> (Unfortunately, there is no repro for those.)
>
> The two crashes seem to share the same issue.
> In both cases, (uninitialized) memory access violation occurs
> when "rdp->cblist" is about to be accessed (kernel/rcu/tree.c:2838,1728).
> I guess those are freed before the use, but I still haven't figured
> out the reason why.
> I'm looking forward to some help.
You lost me on this one. In both cases, rdp references a per-CPU
variable that is implicitly initialized to all zeroes, due to being
(sort of) a C-language global.
If a callback is queued early, then the following lines in __call_rcu()
will make an honest list of that field because of the :
if (rcu_segcblist_empty(&rdp->cblist))
rcu_segcblist_init(&rdp->cblist);
Otherwise, when rcu_init() is invoked during early boot, we have this
in rcu_init_percpu_data(), which is called from rcutree_prepare_cpu()
which is called from rcu_init(), which is called from start_kernel():
if (rcu_segcblist_empty(&rdp->cblist) && /* No early-boot CBs? */
!init_nocb_callback_list(rdp))
rcu_segcblist_init(&rdp->cblist); /* Re-enable callbacks. */
So either init_nocb_callback_list() initializes the alternative callback
lists for a no-CBs CPU or rcu_segcblist_init() again makes an honest
list of that field.
My guess is that your tool is missing the
rdp = this_cpu_ptr(rsp->rda);
in the __call_rcu() case, and also missing the
struct rcu_data *rdp = per_cpu_ptr(rsp->rda, cpu);
Note that the ->rda field is explicitly compile-time initialized to
the base address of the per-CPU variable, which is rcu_preempt_data,
rcu_bh_data, or rcu_sched_data, depending on which RCU flavor is at hand.
(In v4.20-rc1, these are all merged into a single flavor to rule them all.)
Alternatively, your tool might be missing the implicit initialization
of per-CPU variables.
Or maybe I am missing something. If so, please let me know what it is.
Thanx, Paul
> Crash log 1
> =========================================
> BUG: KMSAN: uninit-value in __rcu_process_callbacks
> kernel/rcu/tree.c:2838 [inline]
> BUG: KMSAN: uninit-value in rcu_process_callbacks+0x5ac/0x1cb0
> kernel/rcu/tree.c:2864
> CPU: 0 PID: 20 Comm: kauditd Not tainted 4.19.0-rc8+ #18
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> <IRQ>
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x305/0x460 lib/dump_stack.c:113
> kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
> __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
> __rcu_process_callbacks kernel/rcu/tree.c:2838 [inline]
> rcu_process_callbacks+0x5ac/0x1cb0 kernel/rcu/tree.c:2864
> __do_softirq+0x5ff/0xa55 kernel/softirq.c:292
> invoke_softirq kernel/softirq.c:373 [inline]
> irq_exit+0x22d/0x270 kernel/softirq.c:414
> exiting_irq+0xe/0x10 arch/x86/include/asm/apic.h:536
> smp_apic_timer_interrupt+0x64/0x90 arch/x86/kernel/apic/apic.c:1059
> apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:869
> </IRQ>
> RIP: 0010:finish_lock_switch+0x2b/0x40 kernel/sched/core.c:2578
> Code: 48 89 e5 53 48 89 fb e8 e3 43 9a 00 8b b8 88 0c 00 00 48 8b 00
> 48 85 c0 75 12 48 89 df e8 7d 38 9a 00 c6 00 00 c6 03 00 fb 5b <5d> c3
> e8 de 42 9a 00 eb e7 66 66 66 2e 0f 1f 84 00 00 00 00 00 55
> RSP: 0018:ffff88010622fca0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
> RAX: ffff8801105bcc40 RBX: ffff8801061554c0 RCX: ffff8801105bdc40
> RDX: ffff8801105bdc40 RSI: aaaaaaaaaaaab000 RDI: ffffea00077ec560
> RBP: ffff88010622fca0 R08: ffffffff7fffffff R09: 0000000000000002
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800751cb880
> R13: 0000000000000000 R14: ffff880106155db8 R15: ffff88013fcb9c40
> finish_task_switch+0xe3/0x270 kernel/sched/core.c:2679
> context_switch kernel/sched/core.c:2832 [inline]
> __schedule+0x78f/0x8f0 kernel/sched/core.c:3479
> schedule+0x1cc/0x300 kernel/sched/core.c:3523
> kauditd_thread+0xc64/0xee0 kernel/audit.c:889
> kthread+0x5b1/0x5f0 kernel/kthread.c:247
> ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
>
> Uninit was created at:
> kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
> kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
> kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
> __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
> alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
> alloc_pages include/linux/gfp.h:511 [inline]
> alloc_slab_page mm/slub.c:1459 [inline]
> allocate_slab mm/slub.c:1604 [inline]
> new_slab+0x552/0x1f30 mm/slub.c:1675
> new_slab_objects mm/slub.c:2438 [inline]
> ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
> __slab_alloc mm/slub.c:2630 [inline]
> slab_alloc_node mm/slub.c:2693 [inline]
> slab_alloc mm/slub.c:2735 [inline]
> kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
> kmem_cache_zalloc include/linux/slab.h:697 [inline]
> avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
> avc_update_node+0x172/0x1ee0 security/selinux/avc.c:859
> avc_denied+0x312/0x360 security/selinux/avc.c:1024
> avc_has_perm_noaudit+0x733/0x770 security/selinux/avc.c:1155
> avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
> sock_has_perm security/selinux/hooks.c:4539 [inline]
> selinux_socket_sendmsg+0x297/0x360 security/selinux/hooks.c:4875
> security_socket_sendmsg+0x127/0x200 security/security.c:1410
> sock_sendmsg net/socket.c:628 [inline]
> ___sys_sendmsg+0xd5f/0x1290 net/socket.c:2116
> __sys_sendmsg net/socket.c:2154 [inline]
> __do_sys_sendmsg net/socket.c:2163 [inline]
> __se_sys_sendmsg+0x307/0x460 net/socket.c:2161
> __x64_sys_sendmsg+0x4a/0x70 net/socket.c:2161
> do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
> entry_SYSCALL_64_after_hwframe+0x63/0xe7
> =========================================================
>
> Crash log 2
> =========================================================
> BUG: KMSAN: uninit-value in rcu_accelerate_cbs+0x821/0x990
> kernel/rcu/tree.c:1728
> CPU: 0 PID: 10 Comm: rcu_sched Not tainted 4.19.0-rc8+ #18
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x305/0x460 lib/dump_stack.c:113
> kmsan_report+0x1a2/0x2e0 mm/kmsan/kmsan.c:917
> __msan_warning+0x7d/0xe0 mm/kmsan/kmsan_instr.c:500
> rcu_accelerate_cbs+0x821/0x990 kernel/rcu/tree.c:1728
> __note_gp_changes+0x2ac/0x9e0 kernel/rcu/tree.c:1807
> rcu_gp_cleanup kernel/rcu/tree.c:2109 [inline]
> rcu_gp_kthread+0x3019/0x3990 kernel/rcu/tree.c:2236
> kthread+0x5b1/0x5f0 kernel/kthread.c:247
> ret_from_fork+0x35/0x40 arch/x86/entry/entry_64.S:416
>
> Uninit was created at:
> kmsan_save_stack_with_flags mm/kmsan/kmsan.c:255 [inline]
> kmsan_internal_alloc_meta_for_pages+0x157/0x730 mm/kmsan/kmsan.c:693
> kmsan_alloc_page+0x80/0xe0 mm/kmsan/kmsan_hooks.c:320
> __alloc_pages_nodemask+0x128c/0x69b0 mm/page_alloc.c:4416
> alloc_pages_current+0x51f/0x760 mm/mempolicy.c:2093
> alloc_pages include/linux/gfp.h:511 [inline]
> alloc_slab_page mm/slub.c:1459 [inline]
> allocate_slab mm/slub.c:1604 [inline]
> new_slab+0x552/0x1f30 mm/slub.c:1675
> new_slab_objects mm/slub.c:2438 [inline]
> ___slab_alloc+0x1414/0x1dd0 mm/slub.c:2590
> __slab_alloc mm/slub.c:2630 [inline]
> slab_alloc_node mm/slub.c:2693 [inline]
> slab_alloc mm/slub.c:2735 [inline]
> kmem_cache_alloc+0xc9b/0xda0 mm/slub.c:2740
> kmem_cache_zalloc include/linux/slab.h:697 [inline]
> avc_alloc_node+0x109/0xb90 security/selinux/avc.c:572
> avc_insert security/selinux/avc.c:696 [inline]
> avc_compute_av+0x31e/0x1050 security/selinux/avc.c:1008
> avc_has_perm_noaudit+0x516/0x770 security/selinux/avc.c:1149
> avc_has_perm+0x172/0x480 security/selinux/avc.c:1184
> selinux_socket_create+0x248/0x3c0 security/selinux/hooks.c:4560
> security_socket_create+0x146/0x210 security/security.c:1372
> __sock_create+0x26b/0xf30 net/socket.c:1232
> sock_create net/socket.c:1317 [inline]
> __sys_socket+0x180/0x670 net/socket.c:1347
> __do_sys_socket net/socket.c:1356 [inline]
> __se_sys_socket+0x8d/0xb0 net/socket.c:1354
> __x64_sys_socket+0x4a/0x70 net/socket.c:1354
> do_syscall_64+0xb8/0x100 arch/x86/entry/common.c:291
> entry_SYSCALL_64_after_hwframe+0x63/0xe7
> =========================================================
>
> Thanks,
> Kyungtae Kim
>
