I hit "allowing the OOM killer to select the same thread again" problem ( http://lkml.kernel.org/r/[email protected] ), but I think that there is a bug in down_write_killable() series (at least "locking, rwsem: introduce basis for down_write_killable" patch).
Complete log is at http://I-love.SAKURA.ne.jp/tmp/serial-20160510-sem.txt.xz . ---------- [ 48.303867] Out of memory: Kill process 1314 (tgid=1314) score 1000 or sacrifice child [ 48.308582] Killed process 1314 (tgid=1314) total-vm:70844kB, anon-rss:1980kB, file-rss:0kB, shmem-rss:0kB [ 49.323719] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 49.338146] [ 49.338146] Showing all locks held in the system: (...snipped...) [ 49.801071] 1 lock held by tgid=1314/1314: [ 49.803953] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff810fddac>] acct_collect+0x5c/0x1e0 [ 49.809200] 1 lock held by tgid=1314/1443: [ 49.812102] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81073a45>] do_exit+0x175/0xb10 (...snipped...) [ 51.188928] oom_reaper: unable to reap pid:1443 (tgid=1314) [ 55.576750] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 57.717917] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 59.285880] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 60.818697] oom_reaper: unable to reap pid:1314 (tgid=1314) (...snipped...) [ 174.429572] tgid=1314 D ffff88003ad93b90 0 1314 1209 0x00100084 [ 174.429573] ffff88003ad93b90 ffff88003ad8f6b8 ffff880039c3a140 ffff88003ad8c080 [ 174.429574] ffff88003ad94000 ffff88003ad8f6a0 ffff88003ad8f6b8 0000000000000000 [ 174.429575] 0000000000000008 ffff88003ad93ba8 ffffffff81616190 ffff88003ad8c080 [ 174.429585] Call Trace: [ 174.429586] [<ffffffff81616190>] schedule+0x30/0x80 [ 174.429587] [<ffffffff81619e26>] rwsem_down_read_failed+0xd6/0x140 [ 174.429589] [<ffffffff812dd6f8>] call_rwsem_down_read_failed+0x18/0x30 [ 174.429590] [<ffffffff816196dd>] down_read+0x3d/0x50 [ 174.429592] [<ffffffff810fddac>] ? acct_collect+0x5c/0x1e0 [ 174.429593] [<ffffffff810fddac>] acct_collect+0x5c/0x1e0 [ 174.429594] [<ffffffff81073ff5>] do_exit+0x725/0xb10 [ 174.429594] [<ffffffff81074467>] do_group_exit+0x47/0xc0 [ 174.429596] [<ffffffff8108075f>] get_signal+0x20f/0x7b0 [ 174.429597] [<ffffffff81024fb2>] do_signal+0x32/0x700 [ 174.429598] [<ffffffff810bdc69>] ? trace_hardirqs_on+0x9/0x10 [ 174.429599] [<ffffffff810c3552>] ? rwsem_wake+0x72/0xe0 [ 174.429600] [<ffffffff812dd78b>] ? call_rwsem_wake+0x1b/0x30 [ 174.429601] [<ffffffff810b9ee0>] ? up_read+0x30/0x40 [ 174.429602] [<ffffffff8106b495>] ? exit_to_usermode_loop+0x29/0x9e [ 174.429603] [<ffffffff8106b4bf>] exit_to_usermode_loop+0x53/0x9e [ 174.429604] [<ffffffff8100348d>] prepare_exit_to_usermode+0x7d/0x90 [ 174.429605] [<ffffffff8161bd3e>] retint_user+0x8/0x23 [ 174.429605] tgid=1314 D ffff88003aa2fbd0 0 1443 1209 0x00000084 [ 174.429607] ffff88003aa2fbd0 ffff88003ad8f6b8 ffff8800382060c0 ffff88003aa2a140 [ 174.429608] ffff88003aa30000 ffff88003ad8f6a0 ffff88003ad8f6b8 ffff88003aa2a140 [ 174.429609] 0000000000000008 ffff88003aa2fbe8 ffffffff81616190 ffff88003aa2a140 [ 174.429610] Call Trace: [ 174.429611] [<ffffffff81616190>] schedule+0x30/0x80 [ 174.429612] [<ffffffff81619e26>] rwsem_down_read_failed+0xd6/0x140 [ 174.429613] [<ffffffff810bdb99>] ? trace_hardirqs_on_caller+0xf9/0x1c0 [ 174.429614] [<ffffffff812dd6f8>] call_rwsem_down_read_failed+0x18/0x30 [ 174.429615] [<ffffffff816196dd>] down_read+0x3d/0x50 [ 174.429616] [<ffffffff81073a45>] ? do_exit+0x175/0xb10 [ 174.429616] [<ffffffff81073a45>] do_exit+0x175/0xb10 [ 174.429617] [<ffffffff81074467>] do_group_exit+0x47/0xc0 [ 174.429618] [<ffffffff8108075f>] get_signal+0x20f/0x7b0 [ 174.429619] [<ffffffff81024fb2>] do_signal+0x32/0x700 [ 174.429620] [<ffffffff8161acae>] ? _raw_spin_unlock_irq+0x2e/0x40 [ 174.429621] [<ffffffff8161a2bf>] ? rwsem_down_write_failed_killable+0x1ef/0x280 [ 174.429631] [<ffffffff8106b555>] ? syscall_slow_exit_work+0x4b/0x10d [ 174.429632] [<ffffffff8106b495>] ? exit_to_usermode_loop+0x29/0x9e [ 174.429633] [<ffffffff8106b4bf>] exit_to_usermode_loop+0x53/0x9e [ 174.429634] [<ffffffff81003715>] do_syscall_64+0x135/0x1b0 [ 174.429635] [<ffffffff8161b43f>] entry_SYSCALL64_slow_path+0x25/0x25 (...snipped...) [ 217.651477] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 219.071975] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 220.508961] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 222.022111] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 223.560166] oom_reaper: unable to reap pid:1314 (tgid=1314) [ 225.267750] oom_reaper: unable to reap pid:1314 (tgid=1314) ---------- 2 threads (PID: 1314 and 1443) are sleeping at rwsem_down_read_failed() but no thread is sleeping at rwsem_down_write_failed_killable(). If there is no thread waiting for write lock, threads waiting for read lock must be able to run. This suggests that one of threads which was waiting for write lock forgot to wake up reader threads. Looking at rwsem_down_read_failed(), reader threads waiting for the writer thread to release the lock are waiting on sem->wait_list list. Looking at __rwsem_down_write_failed_common(), when the writer thread escaped the /* Block until there are no active lockers. */ do { if (signal_pending_state(state, current)) { raw_spin_lock_irq(&sem->wait_lock); ret = ERR_PTR(-EINTR); goto out; } schedule(); set_current_state(state); } while ((count = sem->count) & RWSEM_ACTIVE_MASK); loop due to SIGKILL, I think that the writer thread needs to check for remaining threads on sem->wait_list list and wake up reader threads before rwsem_down_write_failed_killable() returns -EINTR.
