On Thu, Sep 27, 2018 at 12:27 PM, Dmitry Vyukov <dvyu...@google.com> wrote: > On Thu, Sep 27, 2018 at 1:35 AM, John Johansen > <john.johan...@canonical.com> wrote: >> On 09/26/2018 02:22 PM, Daniel Borkmann wrote: >>> On 09/26/2018 11:09 PM, Tetsuo Handa wrote: >>>> Hello, Alexei and Daniel. >>>> >>>> Can you show us how to run testcases you are testing? >>> >>> Sorry for the delay; currently quite backlogged but will definitely take a >>> look >>> at these reports. Regarding your question: majority of test cases are in the >>> kernel tree under selftests, see tools/testing/selftests/bpf/ . >>> >> >> Its unlikely to be apparmor. I went through the reports and saw nothing that >> would indicate apparmor involvement, but the primary reason is what is being >> tested >> in upstream apparmor atm. >> >> The current upstream code does nothing directly with skbuffs. Its >> possible that the audit code paths (kernel audit does grab skbuffs) >> could, but there are only a couple cases that would be triggered in >> the current fuzzing so this seems to be an unlikely source for such a >> large leak. > > > Ack. There is no direct evidence against apparmor, I am just trying to > get at least some hooks re the root cause. > > From all the weak indirect evidence, I leaning towards skb allocation > in an infinite loop (or a timer with infinite rate). > >>>> On 2018/09/22 22:25, Tetsuo Handa wrote: >>>>> Hello. >>>>> >>>>> syzbot is reporting many lockup problems on bpf.git / bpf-next.git / >>>>> net.git / net-next.git trees. >>>>> >>>>> INFO: rcu detected stall in br_multicast_port_group_expired (2) >>>>> >>>>> https://syzkaller.appspot.com/bug?id=15c7ad8cf35a07059e8a697a22527e11d294bc94 >>>>> >>>>> INFO: rcu detected stall in tun_chr_close >>>>> >>>>> https://syzkaller.appspot.com/bug?id=6c50618bde03e5a2eefdd0269cf9739c5ebb8270 >>>>> >>>>> INFO: rcu detected stall in discover_timer >>>>> >>>>> https://syzkaller.appspot.com/bug?id=55da031ddb910e58ab9c6853a5784efd94f03b54 >>>>> >>>>> INFO: rcu detected stall in ret_from_fork (2) >>>>> >>>>> https://syzkaller.appspot.com/bug?id=c83129a6683b44b39f5b8864a1325893c9218363 >>>>> >>>>> INFO: rcu detected stall in addrconf_rs_timer >>>>> >>>>> https://syzkaller.appspot.com/bug?id=21c029af65f81488edbc07a10ed20792444711b6 >>>>> >>>>> INFO: rcu detected stall in kthread (2) >>>>> >>>>> https://syzkaller.appspot.com/bug?id=6accd1ed11c31110fed1982f6ad38cc9676477d2 >>>>> >>>>> INFO: rcu detected stall in ext4_filemap_fault >>>>> >>>>> https://syzkaller.appspot.com/bug?id=817e38d20e9ee53390ac361bf0fd2007eaf188af >>>>> >>>>> INFO: rcu detected stall in run_timer_softirq (2) >>>>> >>>>> https://syzkaller.appspot.com/bug?id=f5a230a3ff7822f8d39fddf8485931bd06ae47fe >>>>> >>>>> INFO: rcu detected stall in bpf_prog_ADDR >>>>> >>>>> https://syzkaller.appspot.com/bug?id=fb4911fd0e861171cc55124e209f810a0dd68744 >>>>> >>>>> INFO: rcu detected stall in __run_timers (2) >>>>> >>>>> https://syzkaller.appspot.com/bug?id=65416569ddc8d2feb8f19066aa761f5a47f7451a >>>>> >>>>> The cause of lockup seems to be flood of printk() messages from memory >>>>> allocation >>>>> failures, and one of out_of_memory() messages indicates that >>>>> skbuff_head_cache >>>>> usage is huge enough to suspect in-kernel memory leaks. >>>>> >>>>> [ 1554.547011] skbuff_head_cache 1847887KB 1847887KB >>>>> >>>>> Unfortunately, we cannot find from logs what syzbot is trying to do >>>>> because constant printk() messages is flooding away syzkaller messages. >>>>> Can you try running your testcases with kmemleak enabled? >>>>> >>>> >>>> On 2018/09/27 2:35, Dmitry Vyukov wrote: >>>>> I also started suspecting Apparmor. We switched to Apparmor on Aug 30: >>>>> https://groups.google.com/d/msg/syzkaller-bugs/o73lO4KGh0w/j9pcH2tSBAAJ >>>>> Now the instances that use SELinux and Smack explicitly contain that >>>>> in the name, but the rest are Apparmor. >>>>> Aug 30 roughly matches these assorted "task hung" reports. Perhaps >>>>> some Apparmor hook leaks a reference to skbs? >>>> >>>> Maybe. They have CONFIG_DEFAULT_SECURITY="apparmor". But I'm wondering why >>>> this problem is not occurring on linux-next.git when this problem is >>>> occurring >>>> on bpf.git / bpf-next.git / net.git / net-next.git trees. Is syzbot running >>>> different testcases depending on which git tree is targeted? >>>> >> this is another reason that it is doubtful that its apparmor.
On Thu, Sep 27, 2018 at 2:52 PM, edumazet > Have you tried kmemleak perhaps, it might give us a clue, but it seems > obvious the leak would be in TX path. So, I've tried. Now what? :) I've uploaded all reports to: https://drive.google.com/file/d/107LUW0zmYbXmxfQCWoLpeenxXJsXIkxj/view?usp=sharing This is on net tree d4ce58082f206bf6e7d697380c7bc5480a8b0264 memory leak in __lookup_hash 33 Sep 27 2018 16:35:50 memory leak in new_inode_pseudo 43 Sep 27 2018 16:41:14 memory leak in path_openat 1 Sep 27 2018 16:12:53 memory leak in rhashtable_init 1 Sep 27 2018 16:41:41 memory leak in shmem_symlink 4 Sep 27 2018 16:34:34 memory leak in __anon_vma_prepare 1 Sep 27 2018 17:30:02 memory leak in __do_execve_file 4 Sep 27 2018 18:14:10 memory leak in __do_sys_perf_event_open 2 Sep 27 2018 17:40:05 memory leak in __es_insert_extent 3 Sep 27 2018 17:24:52 memory leak in __getblk_gfp 3 Sep 27 2018 18:19:30 memory leak in __handle_mm_fault 1 Sep 27 2018 18:11:31 memory leak in __hw_addr_create_ex 2 Sep 27 2018 18:10:56 memory leak in __ip_mc_inc_group 13 Sep 27 2018 18:24:31 memory leak in __khugepaged_enter 1 Sep 27 2018 15:40:25 memory leak in __list_lru_init 27 Sep 27 2018 17:30:48 memory leak in __neigh_create 12 Sep 27 2018 17:40:28 memory leak in __netlink_create 1 Sep 27 2018 15:40:23 memory leak in __register_sysctl_table 1 Sep 27 2018 17:36:57 memory leak in __send_signal 1 Sep 27 2018 18:30:48 memory leak in __sys_socket 7 Sep 27 2018 15:43:20 memory leak in anon_inode_getfile 4 Sep 27 2018 18:17:29 memory leak in bpf_prog_store_orig_filter 1 Sep 27 2018 17:59:14 memory leak in br_multicast_new_group 2 Sep 27 2018 18:16:42 memory leak in br_multicast_new_port_group 3 Sep 27 2018 18:17:39 memory leak in build_sched_domains 2 Sep 27 2018 17:28:55 memory leak in clone_mnt 2 Sep 27 2018 18:18:48 memory leak in compute_effective_progs 1 Sep 27 2018 18:35:15 memory leak in create_empty_buffers 1 Sep 27 2018 17:35:42 memory leak in create_filter_start 2 Sep 27 2018 18:16:00 memory leak in create_pipe_files 3 Sep 27 2018 18:21:47 memory leak in do_ip6t_set_ctl 1 Sep 27 2018 15:40:21 memory leak in do_ipt_set_ctl 1 Sep 27 2018 17:37:35 memory leak in do_signalfd4 1 Sep 27 2018 18:00:45 memory leak in do_syslog 2 Sep 27 2018 18:06:22 memory leak in ep_insert 11 Sep 27 2018 17:39:14 memory leak in ep_ptable_queue_proc 3 Sep 27 2018 17:37:17 memory leak in ext4_mb_new_group_pa 1 Sep 27 2018 17:36:04 memory leak in ext4_mb_new_inode_pa 3 Sep 27 2018 17:39:19 memory leak in fdb_create 13 Sep 27 2018 18:24:03 memory leak in fib6_add_1 11 Sep 27 2018 18:19:53 memory leak in fib_table_insert 1 Sep 27 2018 17:36:49 memory leak in find_get_context 1 Sep 27 2018 15:42:00 memory leak in fsnotify_add_mark_locked 11 Sep 27 2018 17:31:41 memory leak in idr_get_free 2 Sep 27 2018 18:17:34 memory leak in iget_locked 1 Sep 27 2018 17:52:59 memory leak in inet_frag_find 3 Sep 27 2018 18:23:53 memory leak in inotify_update_watch 25 Sep 27 2018 17:30:59 memory leak in ioc_create_icq 1 Sep 27 2018 15:42:40 memory leak in ip6_pol_route 4 Sep 27 2018 18:24:59 memory leak in ip6_route_info_create 1 Sep 27 2018 17:38:09 memory leak in ip6t_register_table 1 Sep 27 2018 17:35:39 memory leak in ip_route_output_key_hash_rcu 4 Sep 27 2018 18:21:21 memory leak in ipt_register_table 3 Sep 27 2018 17:39:58 memory leak in ipv6_add_addr 1 Sep 27 2018 17:38:49 memory leak in load_elf_binary 1 Sep 27 2018 18:09:44 memory leak in map_create 9 Sep 27 2018 18:25:10 memory leak in memcg_update_all_list_lrus 1 Sep 27 2018 15:39:36 memory leak in ndisc_send_rs 1 Sep 27 2018 17:50:03 memory leak in neigh_table_init 6 Sep 27 2018 17:23:41 memory leak in nf_hook_entries_grow 5 Sep 27 2018 15:43:41 memory leak in packet_sendmsg 1 Sep 27 2018 18:08:43 memory leak in pcpu_create_chunk 1 Sep 27 2018 17:48:41 memory leak in prepare_creds 18 Sep 27 2018 17:29:10 memory leak in prepare_kernel_cred 15 Sep 27 2018 18:23:42 memory leak in process_preds 2 Sep 27 2018 18:11:01 memory leak in rht_deferred_worker 9 Sep 27 2018 18:24:29 memory leak in sched_init_domains 2 Sep 27 2018 18:12:42 memory leak in sctp_addr_wq_mgmt 1 Sep 27 2018 17:46:20 memory leak in sget 5 Sep 27 2018 18:03:40 memory leak in shmem_symlink 30 Sep 27 2018 17:32:09 memory leak in skb_clone 3 Sep 27 2018 18:13:22 memory leak in submit_bh_wbc 1 Sep 27 2018 17:49:06 memory leak in tracepoint_probe_register_prio 1 Sep 27 2018 17:39:13 memory leak in xt_replace_table 4 Sep 27 2018 15:43:19 memory leak in __delayacct_tsk_init 2 Sep 27 2018 17:02:53 memory leak in disk_expand_part_tbl 1 Sep 27 2018 16:59:05 memory leak in do_ip6t_set_ctl 14 Sep 27 2018 15:46:37 memory leak in neigh_table_init 4 Sep 27 2018 17:10:29 memory leak in do_check 1 Sep 27 2018 18:38:13
