On 24/06/24(Mon) 22:32, Dana Koch wrote:
> Dana Koch <d...@google.com> schrieb am So., 23. Juni 2024, 19:50:
>
> > > Could you try the diff below? Stuart confirmed it prevents the hang on
> > > his machine.
> >
> > This also seems to be working well for me so far.
> >
>
> Okay, I've got an actual panic now, with similar reproduction steps
> building LLVM instead of the kernel; not sure if this is related or
> unrelated to the underlying problem, but here's a ddb session.
It's unrelated. The panic happens because as soon as a CPU enters ddb
the locking is bypassed before all CPUs got parked.
The issue has been identified. It's a starvation due to assumptions
made in userland by the use of sched_yield(2).
> I also remembered `show witness` which may or may not be helpful here,
> pointing at a lock order reversal.
>
>
> __mp_lock_spin: 0xffffff8001328528 lock spun out
> Stopped at __mp_lock+0x138: ldr w8, [x23,#1352]
> ddb{17}> __mp_lock_spin: 0xffffff8001328528 lock spun out
> panic: kernel diagnostic assertion "uvm_page_owner_locked_p(pg)" failed:
> file "
> /home/dana/src/openbsd/openbsd-src/sys/uvm/uvm_page.c", line 1268
>
> ddb{17}> trace__mp_lock_spin: 0xffffff8001328528 lock spun out
>
> db_enter() at __mp_lock+0x134
> __mp_lock() at svc_handler+0x42c
> svc_handler() at do_el0_sync+0xc8
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at 0x46d2d0fe0
> --- trap ---
> end of kernel
> ddb{17}> ps /o
> TID PID UID PRFLAGS PFLAGS CPU COMMAND
> 483459 1841 1000 0x18000003 0 4 c++
> *507682 54858 1000 0x18000003 0 17 c++
> 491941 81792 1000 0x18000003 0 19 c++
> 92517 49319 1000 0x18000003 0 23 c++
> 479846 23280 1000 0x18000003 0 6 c++
> 200584 12281 1000 0x18000003 0 22 c++
> 421381 35598 1000 0x18000003 0 5 c++
> 44090 50287 1000 0x18000003 0 0 c++
> 188893 26365 1000 0x18000003 0 18 c++
> 74859 4393 1000 0x18000003 0 7 c++
> 208648 45008 1000 0x18000003 0 8 c++
> 97267 77502 1000 0x18000003 0 16 c++
> 174351 42043 1000 0x18000003 0 15 c++
> 501477 35285 1000 0x18000003 0 12 c++
> 255809 23573 1000 0x18000003 0 21 c++
> 106102 27978 1000 0x18000003 0 14 c++
> 294101 98084 1000 0x18000003 0 13 c++
> 76692 99593 1000 0x18000003 0 20 c++
> 520521 17740 1000 0x18000003 0 9 c++
> 343763 76972 1000 0x18000003 0 11 c++
> 372390 54052 1000 0x18000003 0 3 c++
> 435277 88527 1000 0x18000003 0 10 c++
> 99868 81397 0 0x14000 0x200 2 reaper
> 321683 8245 0 0x14000 0x200 1K pagedaemon
> ddb{17}> show all locks
> CPU 1:
> exclusive mutex &pmap->pm_mtx r = 0 (0xffffff816a2cbc48)
> exclusive mutex &uvm.pageqlock r = 0 (0xffffff80012c5958)
> Process 1841 (c++) thread 0xffffff8165132628 (483459)
> exclusive rwlock amaplk r = 0 (0xffffff81654b01d8)
> shared rwlock vmmaplk r = 0 (0xffffff816e1727e0)
> Process 81792 (c++) thread 0xffffff8165133558 (491941)
> exclusive rwlock amaplk r = 0 (0xffffff8176c011d8)
> shared rwlock vmmaplk r = 0 (0xffffff809fe2ab48)
> Process 49319 (c++) thread 0xffffff816d089570 (92517)
> exclusive rwlock amaplk r = 0 (0xffffff816cb97c50)
> shared rwlock vmmaplk r = 0 (0xffffff816ff25618)
> Process 23280 (c++) thread 0xffffff816d088b50 (479846)
> exclusive rwlock amaplk r = 0 (0xffffff8176c104e0)
> shared rwlock vmmaplk r = 0 (0xffffff8167d8ab58)
> Process 12281 (c++) thread 0xffffff816d0888c8 (200584)
> exclusive rwlock amaplk r = 0 (0xffffff8176c100f0)
> shared rwlock vmmaplk r = 0 (0xffffff809f81f460)
> Process 35598 (c++) thread 0xffffff816d0883b8 (421381)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10e10)
> shared rwlock vmmaplk r = 0 (0xffffff816ff25eb0)
> Process 50287 (c++) thread 0xffffff816d089060 (44090)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10c88)
> shared rwlock vmmaplk r = 0 (0xffffff816ff252a8)
> Process 26365 (c++) thread 0xffffff816d0897f8 (188893)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10eb8)
> shared rwlock vmmaplk r = 0 (0xffffff8176112b58)
> Process 4393 (c++) thread 0xffffff8165133048 (74859)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10860)
> shared rwlock vmmaplk r = 0 (0xffffff8176112ec8)
> Process 45008 (c++) thread 0xffffff8165132dc0 (208648)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10320)
> shared rwlock vmmaplk r = 0 (0xffffff81761127e8)
> Process 77502 (c++) thread 0xffffff816ab95d00 (97267)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10048)
> shared rwlock vmmaplk r = 0 (0xffffff81761129a0)
> Process 42043 (c++) thread 0xffffff816ab95568 (174351)
> exclusive rwlock amaplk r = 0 (0xffffff816a3a2f68)
> shared rwlock vmmaplk r = 0 (0xffffff8176112d10)
> Process 35285 (c++) thread 0xffffff81651343a8 (501477)
> exclusive rwlock amaplk r = 0 (0xffffff8176c105c0)
> shared rwlock vmmaplk r = 0 (0xffffff816eebcb48)
> Process 23573 (c++) thread 0xffffff8165135560 (255809)
> exclusive rwlock amaplk r = 0 (0xffffff8176c109e8)
> shared rwlock vmmaplk r = 0 (0xffffff816eebc468)
> Process 27978 (c++) thread 0xffffff8165135cf8 (106102)
> exclusive rwlock amaplk r = 0 (0xffffff816da1acf8)
> shared rwlock vmmaplk r = 0 (0xffffff816eebcd00)
> Process 98084 (c++) thread 0xffffff81651352d8 (294101)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10080)
> shared rwlock vmmaplk r = 0 (0xffffff816e172ec0)
> Process 99593 (c++) thread 0xffffff8165132b38 (76692)
> exclusive rwlock amaplk r = 0 (0xffffff816da1a7f0)
> shared rwlock vmmaplk r = 0 (0xffffff816e172628)
> Process 17740 (c++) thread 0xffffff816371e2b8 (520521)
> exclusive rwlock amaplk r = 0 (0xffffff8176c109b0)
> shared rwlock vmmaplk r = 0 (0xffffff8167d8a630)
> Process 76972 (c++) thread 0xffffff82c5df3708 (343763)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10240)
> shared rwlock vmmaplk r = 0 (0xffffff8167d8ad10)
> Process 54052 (c++) thread 0xffffff816ab94638 (372390)
> exclusive rwlock amaplk r = 0 (0xffffff8176c10cc0)
> shared rwlock vmmaplk r = 0 (0xffffff809f7bbd00)
> Process 88527 (c++) thread 0xffffff816ab95a78 (435277)
> exclusive rwlock amaplk r = 0 (0xffffff81704a4590)
> Process 81397 (reaper) thread 0xffffff8162de8f48 (99868)
> exclusive rwlock amaplk r = 0 (0xffffff8176246780)
> Process 8245 (pagedaemon) thread 0xffffff8162de91d0 (321683)
> exclusive rwlock amaplk r = 0 (0xffffff816f569f28)
> exclusive kernel_lock &kernel_lock r = 0 (0xffffff8001328d30)
> ddb{17}> mach ddbcpu 0
> Stopped at __mp_lock+0x138: ldr w8, [x23,#1352]
> TID PID UID PRFLAGS PFLAGS CPU COMMAND
> 483459 1841 1000 0x18000003 0 4 c++
> 507682 54858 1000 0x18000003 0 17 c++
> 491941 81792 1000 0x18000003 0 19 c++
> 92517 49319 1000 0x18000003 0 23 c++
> 479846 23280 1000 0x18000003 0 6 c++
> 200584 12281 1000 0x18000003 0 22 c++
> 421381 35598 1000 0x18000003 0 5 c++
> * 44090 50287 1000 0x18000003 0 0 c++
> 188893 26365 1000 0x18000003 0 18 c++
> 74859 4393 1000 0x18000003 0 7 c++
> 208648 45008 1000 0x18000003 0 8 c++
> 97267 77502 1000 0x18000003 0 16 c++
> 174351 42043 1000 0x18000003 0 15 c++
> 501477 35285 1000 0x18000003 0 12 c++
> 255809 23573 1000 0x18000003 0 21 c++
> 106102 27978 1000 0x18000003 0 14 c++
> 294101 98084 1000 0x18000003 0 13 c++
> 76692 99593 1000 0x18000003 0 20 c++
> 520521 17740 1000 0x18000003 0 9 c++
> 343763 76972 1000 0x18000003 0 11 c++
> 372390 54052 1000 0x18000003 0 3 c++
> 435277 88527 1000 0x18000003 0 10 c++
> 99868 81397 0 0x14000 0x200 2 reaper
> 321683 8245 0 0x14000 0x200 1K pagedaemon
> db_enter() at __mp_lock+0x134
> __mp_lock() at aplintc_irq_handler+0x158
> aplintc_irq_handler() at arm_cpu_irq+0x34
> arm_cpu_irq() at handle_el1h_irq+0x68
> handle_el1h_irq() at db_enter_ddb+0x25c
> db_enter_ddb() at kdb_trap+0x64
> kdb_trap() at db_trapper+0x30
> https://www.openbsd.org/ddb.html describes the minimum info required in bug
> reports. Insufficient info makes it difficult to find and fix bugs.
> ddb{0}> trace
> db_enter() at __mp_lock+0x134
> __mp_lock() at aplintc_irq_handler+0x158
> aplintc_irq_handler() at arm_cpu_irq+0x34
> arm_cpu_irq() at handle_el1h_irq+0x68
> handle_el1h_irq() at db_enter_ddb+0x25c
> db_enter_ddb() at kdb_trap+0x64
> kdb_trap() at db_trapper+0x30
> db_trapper() at handle_el1h_sync+0x68
> handle_el1h_sync() at db_enter+0x14
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at aq_lookup+0x80
> aq_intr() at arm_cpu_irq+0x34
> arm_cpu_irq() at handle_el1h_irq+0x68
> handle_el1h_irq() at do_el1h_sync+0x24
> do_el1h_sync() at handle_el1h_sync+0x68
> handle_el1h_sync() at db_enter+0x14
> db_enter() at __mp_lock+0x134
> __mp_lock() at softintr_biglock_wrap+0x14
> softintr_biglock_wrap() at softintr_dispatch+0x84
> softintr_dispatch() at arm_do_pending_intr+0xfc
> arm_do_pending_intr() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x573a8d4
> --- trap ---
> end of kernel
> ddb{0}> mach ddbcpu 1
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at do_el1h_sync+0x24
> do_el1h_sync() at handle_el1h_sync+0x68
> handle_el1h_sync() at db_enter+0x14
> db_enter() at panic+0x148
> ddb{1}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at do_el1h_sync+0x24
> do_el1h_sync() at handle_el1h_sync+0x68
> handle_el1h_sync() at db_enter+0x14
> db_enter() at panic+0x148
> panic() at __assert+0x28
> panic() at uvm_pagedeactivate+0x1cc
> uvm_pagedeactivate() at uvmpd_scan+0x208
> uvmpd_scan() at uvm_pageout+0x2ac
> uvm_pageout() at proc_trampoline+0xc
> ddb{1}> mach ddbcpu 2
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_anfree_list+0x68
> uvm_anfree_list() at amap_wipeout+0xf8
> amap_wipeout() at uvm_unmap_detach+0x70
> ddb{2}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_anfree_list+0x68
> uvm_anfree_list() at amap_wipeout+0xf8
> amap_wipeout() at uvm_unmap_detach+0x70
> uvm_unmap_detach() at uvm_map_teardown+0x1a8
> uvm_map_teardown() at uvmspace_free+0x70
> uvmspace_free() at reaper+0x128
> reaper() at proc_trampoline+0xc
> ddb{2}> mach ddbcpu 3
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{3}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at 0x4e0de3264
> --- trap ---
> end of kernel
> ddb{3}> mach ddbcpu 4
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_upper+0x2a0
> uvm_fault_upper() at uvm_fault+0xb8
> uvm_fault() at udata_abort+0x138
> ddb{4}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_upper+0x2a0
> uvm_fault_upper() at uvm_fault+0xb8
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x40c4134
> --- trap ---
> end of kernel
> ddb{4}> mach ddbcpu 5
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{5}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5702e90
> --- trap ---
> end of kernel
> ddb{5}> mach ddbcpu 6
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{6}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x570296c
> --- trap ---
> end of kernel
> ddb{6}> mach ddbcpu 7
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{7}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5999d20
> --- trap ---
> end of kernel
> ddb{7}> mach ddbcpu 8
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{8}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5914b74
> --- trap ---
> end of kernel
> ddb{8}> mach ddbcpu 9
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{9}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5914b74
> --- trap ---
> end of kernel
> ddb{9}> mach ddbcpu 0xa
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_anfree_list+0x68
> uvm_anfree_list() at amap_wiperange_chunk+0x100
> amap_wiperange_chunk() at amap_wiperange+0x18c
> ddb{10}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_anfree_list+0x68
> uvm_anfree_list() at amap_wiperange_chunk+0x100
> amap_wiperange_chunk() at amap_wiperange+0x18c
> amap_wiperange() at amap_pp_adjref+0x28c
> amap_pp_adjref() at amap_adjref_anons+0xf0
> amap_adjref_anons() at uvm_unmap_detach+0x70
> uvm_unmap_detach() at sys_munmap+0x128
> sys_munmap() at svc_handler+0x478
> svc_handler() at do_el0_sync+0xc8
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at 0x4f73673c8
> --- trap ---
> end of kernel
> ddb{10}> mach ddbcpu 0xb
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{11}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x59245b4
> --- trap ---
> end of kernel
> ddb{11}> mach ddbcpu 0xc
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{12}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5914b74
> --- trap ---
> end of kernel
> ddb{12}> mach ddbcpu 0xd
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{13}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x4fdc1c0
> --- trap ---
> end of kernel
> ddb{13}> mach ddbcpu 0xe
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{14}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x57038b8
> --- trap ---
> end of kernel
> ddb{14}> mach ddbcpu 0xf
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{15}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5914b1c
> --- trap ---
> end of kernel
> ddb{15}> mach ddbcpu 0x10
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{16}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x4e63188
> --- trap ---
> end of kernel
> ddb{16}> mach ddbcpu 0x11
> Stopped at __mp_lock+0x138: ldr w8, [x23,#1352]
> db_enter() at __mp_lock+0x134
> __mp_lock() at svc_handler+0x42c
> svc_handler() at do_el0_sync+0xc8
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at 0x46d2d0fe0
> --- trap ---
> end of kernel
> ddb{17}> trace
> db_enter() at __mp_lock+0x134
> __mp_lock() at svc_handler+0x42c
> svc_handler() at do_el0_sync+0xc8
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at 0x46d2d0fe0
> --- trap ---
> end of kernel
> ddb{17}> mach ddbcpu 0x12
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{18}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x5933dd8
> --- trap ---
> end of kernel
> ddb{18}> mach ddbcpu 0x13
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{19}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at 0x4819ddc40
> --- trap ---
> end of kernel
> ddb{19}> mach ddbcpu 0x14
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{20}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x58958ac
> --- trap ---
> end of kernel
> ddb{20}> mach ddbcpu 0x15
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{21}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x4e63188
> --- trap ---
> end of kernel
> ddb{21}> mach ddbcpu 0x16
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{22}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xec
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x594b754
> --- trap ---
> end of kernel
> ddb{22}> mach ddbcpu 0x17
> Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> <aplin
> tc_fiq_handler+0x7c>
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> ddb{23}> trace
> db_enter() at aplintc_fiq_handler+0x6c
> aplintc_fiq_handler() at arm_cpu_fiq+0x34
> arm_cpu_fiq() at handle_el1h_fiq+0x68
> handle_el1h_fiq() at mtx_enter+0xe4
> mtx_enter() at uvm_fault_lower+0x630
> uvm_fault_lower() at uvm_fault+0x174
> uvm_fault() at udata_abort+0x138
> udata_abort() at do_el0_sync+0x130
> do_el0_sync() at handle_el0_sync+0x70
> handle_el0_sync() at __ALIGN_SIZE+0x57038b8
> --- trap ---
> end of kernel
> ddb{23}>
> ddb{23}> show witness /b
> Number of known direct relationships is 481
>
> Lock order reversal between "&mp->mnt_lock"(rwlock) and
> "&ip->i_lock"(rrwlock)!
>
> witness: incomplete path, depth 4
> ddb{23}> show panic
> *cpu1: kernel diagnostic assertion "uvm_page_owner_locked_p(pg)" failed:
> file "
> /home/dana/src/openbsd/openbsd-src/sys/uvm/uvm_page.c", line 1268
