On Tue, Jun 25, 2024 at 11:21:52AM +0200, Martin Pieuchot wrote:
> On 24/06/24(Mon) 22:32, Dana Koch wrote:
> > Dana Koch <d...@google.com> schrieb am So., 23. Juni 2024, 19:50:
> >
> > > > Could you try the diff below? Stuart confirmed it prevents the hang on
> > > > his machine.
> > >
> > > This also seems to be working well for me so far.
> > >
> >
> > Okay, I've got an actual panic now, with similar reproduction steps
> > building LLVM instead of the kernel; not sure if this is related or
> > unrelated to the underlying problem, but here's a ddb session.
>
> It's unrelated. The panic happens because as soon as a CPU enters ddb
> the locking is bypassed before all CPUs got parked.
>
> The issue has been identified. It's a starvation due to assumptions
> made in userland by the use of sched_yield(2).
This is a different issue of the same kind. There is not a single
sched_yield(2) in the backtrace. The KERNEL_LOCK spun out because of
some other lock holding up everything.
Maybe this time uvm.pageqlock is so contended that this happens.
In general the issue has been identified, mutexes on arm64 suck.
Using MP_LOCKDEBUG on arm64 will trigger too often on false positives.
> > I also remembered `show witness` which may or may not be helpful here,
> > pointing at a lock order reversal.
> >
> >
> > __mp_lock_spin: 0xffffff8001328528 lock spun out
> > Stopped at __mp_lock+0x138: ldr w8, [x23,#1352]
> > ddb{17}> __mp_lock_spin: 0xffffff8001328528 lock spun out
> > panic: kernel diagnostic assertion "uvm_page_owner_locked_p(pg)" failed:
> > file "
> > /home/dana/src/openbsd/openbsd-src/sys/uvm/uvm_page.c", line 1268
> >
> > ddb{17}> trace__mp_lock_spin: 0xffffff8001328528 lock spun out
> >
> > db_enter() at __mp_lock+0x134
> > __mp_lock() at svc_handler+0x42c
> > svc_handler() at do_el0_sync+0xc8
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at 0x46d2d0fe0
> > --- trap ---
> > end of kernel
> > ddb{17}> ps /o
> > TID PID UID PRFLAGS PFLAGS CPU COMMAND
> > 483459 1841 1000 0x18000003 0 4 c++
> > *507682 54858 1000 0x18000003 0 17 c++
> > 491941 81792 1000 0x18000003 0 19 c++
> > 92517 49319 1000 0x18000003 0 23 c++
> > 479846 23280 1000 0x18000003 0 6 c++
> > 200584 12281 1000 0x18000003 0 22 c++
> > 421381 35598 1000 0x18000003 0 5 c++
> > 44090 50287 1000 0x18000003 0 0 c++
> > 188893 26365 1000 0x18000003 0 18 c++
> > 74859 4393 1000 0x18000003 0 7 c++
> > 208648 45008 1000 0x18000003 0 8 c++
> > 97267 77502 1000 0x18000003 0 16 c++
> > 174351 42043 1000 0x18000003 0 15 c++
> > 501477 35285 1000 0x18000003 0 12 c++
> > 255809 23573 1000 0x18000003 0 21 c++
> > 106102 27978 1000 0x18000003 0 14 c++
> > 294101 98084 1000 0x18000003 0 13 c++
> > 76692 99593 1000 0x18000003 0 20 c++
> > 520521 17740 1000 0x18000003 0 9 c++
> > 343763 76972 1000 0x18000003 0 11 c++
> > 372390 54052 1000 0x18000003 0 3 c++
> > 435277 88527 1000 0x18000003 0 10 c++
> > 99868 81397 0 0x14000 0x200 2 reaper
> > 321683 8245 0 0x14000 0x200 1K pagedaemon
> > ddb{17}> show all locks
> > CPU 1:
> > exclusive mutex &pmap->pm_mtx r = 0 (0xffffff816a2cbc48)
> > exclusive mutex &uvm.pageqlock r = 0 (0xffffff80012c5958)
> > Process 1841 (c++) thread 0xffffff8165132628 (483459)
> > exclusive rwlock amaplk r = 0 (0xffffff81654b01d8)
> > shared rwlock vmmaplk r = 0 (0xffffff816e1727e0)
> > Process 81792 (c++) thread 0xffffff8165133558 (491941)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c011d8)
> > shared rwlock vmmaplk r = 0 (0xffffff809fe2ab48)
> > Process 49319 (c++) thread 0xffffff816d089570 (92517)
> > exclusive rwlock amaplk r = 0 (0xffffff816cb97c50)
> > shared rwlock vmmaplk r = 0 (0xffffff816ff25618)
> > Process 23280 (c++) thread 0xffffff816d088b50 (479846)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c104e0)
> > shared rwlock vmmaplk r = 0 (0xffffff8167d8ab58)
> > Process 12281 (c++) thread 0xffffff816d0888c8 (200584)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c100f0)
> > shared rwlock vmmaplk r = 0 (0xffffff809f81f460)
> > Process 35598 (c++) thread 0xffffff816d0883b8 (421381)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10e10)
> > shared rwlock vmmaplk r = 0 (0xffffff816ff25eb0)
> > Process 50287 (c++) thread 0xffffff816d089060 (44090)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10c88)
> > shared rwlock vmmaplk r = 0 (0xffffff816ff252a8)
> > Process 26365 (c++) thread 0xffffff816d0897f8 (188893)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10eb8)
> > shared rwlock vmmaplk r = 0 (0xffffff8176112b58)
> > Process 4393 (c++) thread 0xffffff8165133048 (74859)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10860)
> > shared rwlock vmmaplk r = 0 (0xffffff8176112ec8)
> > Process 45008 (c++) thread 0xffffff8165132dc0 (208648)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10320)
> > shared rwlock vmmaplk r = 0 (0xffffff81761127e8)
> > Process 77502 (c++) thread 0xffffff816ab95d00 (97267)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10048)
> > shared rwlock vmmaplk r = 0 (0xffffff81761129a0)
> > Process 42043 (c++) thread 0xffffff816ab95568 (174351)
> > exclusive rwlock amaplk r = 0 (0xffffff816a3a2f68)
> > shared rwlock vmmaplk r = 0 (0xffffff8176112d10)
> > Process 35285 (c++) thread 0xffffff81651343a8 (501477)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c105c0)
> > shared rwlock vmmaplk r = 0 (0xffffff816eebcb48)
> > Process 23573 (c++) thread 0xffffff8165135560 (255809)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c109e8)
> > shared rwlock vmmaplk r = 0 (0xffffff816eebc468)
> > Process 27978 (c++) thread 0xffffff8165135cf8 (106102)
> > exclusive rwlock amaplk r = 0 (0xffffff816da1acf8)
> > shared rwlock vmmaplk r = 0 (0xffffff816eebcd00)
> > Process 98084 (c++) thread 0xffffff81651352d8 (294101)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10080)
> > shared rwlock vmmaplk r = 0 (0xffffff816e172ec0)
> > Process 99593 (c++) thread 0xffffff8165132b38 (76692)
> > exclusive rwlock amaplk r = 0 (0xffffff816da1a7f0)
> > shared rwlock vmmaplk r = 0 (0xffffff816e172628)
> > Process 17740 (c++) thread 0xffffff816371e2b8 (520521)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c109b0)
> > shared rwlock vmmaplk r = 0 (0xffffff8167d8a630)
> > Process 76972 (c++) thread 0xffffff82c5df3708 (343763)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10240)
> > shared rwlock vmmaplk r = 0 (0xffffff8167d8ad10)
> > Process 54052 (c++) thread 0xffffff816ab94638 (372390)
> > exclusive rwlock amaplk r = 0 (0xffffff8176c10cc0)
> > shared rwlock vmmaplk r = 0 (0xffffff809f7bbd00)
> > Process 88527 (c++) thread 0xffffff816ab95a78 (435277)
> > exclusive rwlock amaplk r = 0 (0xffffff81704a4590)
> > Process 81397 (reaper) thread 0xffffff8162de8f48 (99868)
> > exclusive rwlock amaplk r = 0 (0xffffff8176246780)
> > Process 8245 (pagedaemon) thread 0xffffff8162de91d0 (321683)
> > exclusive rwlock amaplk r = 0 (0xffffff816f569f28)
> > exclusive kernel_lock &kernel_lock r = 0 (0xffffff8001328d30)
> > ddb{17}> mach ddbcpu 0
> > Stopped at __mp_lock+0x138: ldr w8, [x23,#1352]
> > TID PID UID PRFLAGS PFLAGS CPU COMMAND
> > 483459 1841 1000 0x18000003 0 4 c++
> > 507682 54858 1000 0x18000003 0 17 c++
> > 491941 81792 1000 0x18000003 0 19 c++
> > 92517 49319 1000 0x18000003 0 23 c++
> > 479846 23280 1000 0x18000003 0 6 c++
> > 200584 12281 1000 0x18000003 0 22 c++
> > 421381 35598 1000 0x18000003 0 5 c++
> > * 44090 50287 1000 0x18000003 0 0 c++
> > 188893 26365 1000 0x18000003 0 18 c++
> > 74859 4393 1000 0x18000003 0 7 c++
> > 208648 45008 1000 0x18000003 0 8 c++
> > 97267 77502 1000 0x18000003 0 16 c++
> > 174351 42043 1000 0x18000003 0 15 c++
> > 501477 35285 1000 0x18000003 0 12 c++
> > 255809 23573 1000 0x18000003 0 21 c++
> > 106102 27978 1000 0x18000003 0 14 c++
> > 294101 98084 1000 0x18000003 0 13 c++
> > 76692 99593 1000 0x18000003 0 20 c++
> > 520521 17740 1000 0x18000003 0 9 c++
> > 343763 76972 1000 0x18000003 0 11 c++
> > 372390 54052 1000 0x18000003 0 3 c++
> > 435277 88527 1000 0x18000003 0 10 c++
> > 99868 81397 0 0x14000 0x200 2 reaper
> > 321683 8245 0 0x14000 0x200 1K pagedaemon
> > db_enter() at __mp_lock+0x134
> > __mp_lock() at aplintc_irq_handler+0x158
> > aplintc_irq_handler() at arm_cpu_irq+0x34
> > arm_cpu_irq() at handle_el1h_irq+0x68
> > handle_el1h_irq() at db_enter_ddb+0x25c
> > db_enter_ddb() at kdb_trap+0x64
> > kdb_trap() at db_trapper+0x30
> > https://www.openbsd.org/ddb.html describes the minimum info required in bug
> > reports. Insufficient info makes it difficult to find and fix bugs.
> > ddb{0}> trace
> > db_enter() at __mp_lock+0x134
> > __mp_lock() at aplintc_irq_handler+0x158
> > aplintc_irq_handler() at arm_cpu_irq+0x34
> > arm_cpu_irq() at handle_el1h_irq+0x68
> > handle_el1h_irq() at db_enter_ddb+0x25c
> > db_enter_ddb() at kdb_trap+0x64
> > kdb_trap() at db_trapper+0x30
> > db_trapper() at handle_el1h_sync+0x68
> > handle_el1h_sync() at db_enter+0x14
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at aq_lookup+0x80
> > aq_intr() at arm_cpu_irq+0x34
> > arm_cpu_irq() at handle_el1h_irq+0x68
> > handle_el1h_irq() at do_el1h_sync+0x24
> > do_el1h_sync() at handle_el1h_sync+0x68
> > handle_el1h_sync() at db_enter+0x14
> > db_enter() at __mp_lock+0x134
> > __mp_lock() at softintr_biglock_wrap+0x14
> > softintr_biglock_wrap() at softintr_dispatch+0x84
> > softintr_dispatch() at arm_do_pending_intr+0xfc
> > arm_do_pending_intr() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x573a8d4
> > --- trap ---
> > end of kernel
> > ddb{0}> mach ddbcpu 1
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at do_el1h_sync+0x24
> > do_el1h_sync() at handle_el1h_sync+0x68
> > handle_el1h_sync() at db_enter+0x14
> > db_enter() at panic+0x148
> > ddb{1}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at do_el1h_sync+0x24
> > do_el1h_sync() at handle_el1h_sync+0x68
> > handle_el1h_sync() at db_enter+0x14
> > db_enter() at panic+0x148
> > panic() at __assert+0x28
> > panic() at uvm_pagedeactivate+0x1cc
> > uvm_pagedeactivate() at uvmpd_scan+0x208
> > uvmpd_scan() at uvm_pageout+0x2ac
> > uvm_pageout() at proc_trampoline+0xc
> > ddb{1}> mach ddbcpu 2
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_anfree_list+0x68
> > uvm_anfree_list() at amap_wipeout+0xf8
> > amap_wipeout() at uvm_unmap_detach+0x70
> > ddb{2}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_anfree_list+0x68
> > uvm_anfree_list() at amap_wipeout+0xf8
> > amap_wipeout() at uvm_unmap_detach+0x70
> > uvm_unmap_detach() at uvm_map_teardown+0x1a8
> > uvm_map_teardown() at uvmspace_free+0x70
> > uvmspace_free() at reaper+0x128
> > reaper() at proc_trampoline+0xc
> > ddb{2}> mach ddbcpu 3
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{3}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at 0x4e0de3264
> > --- trap ---
> > end of kernel
> > ddb{3}> mach ddbcpu 4
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_upper+0x2a0
> > uvm_fault_upper() at uvm_fault+0xb8
> > uvm_fault() at udata_abort+0x138
> > ddb{4}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_upper+0x2a0
> > uvm_fault_upper() at uvm_fault+0xb8
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x40c4134
> > --- trap ---
> > end of kernel
> > ddb{4}> mach ddbcpu 5
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{5}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5702e90
> > --- trap ---
> > end of kernel
> > ddb{5}> mach ddbcpu 6
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{6}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x570296c
> > --- trap ---
> > end of kernel
> > ddb{6}> mach ddbcpu 7
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{7}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5999d20
> > --- trap ---
> > end of kernel
> > ddb{7}> mach ddbcpu 8
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{8}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5914b74
> > --- trap ---
> > end of kernel
> > ddb{8}> mach ddbcpu 9
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{9}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5914b74
> > --- trap ---
> > end of kernel
> > ddb{9}> mach ddbcpu 0xa
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_anfree_list+0x68
> > uvm_anfree_list() at amap_wiperange_chunk+0x100
> > amap_wiperange_chunk() at amap_wiperange+0x18c
> > ddb{10}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_anfree_list+0x68
> > uvm_anfree_list() at amap_wiperange_chunk+0x100
> > amap_wiperange_chunk() at amap_wiperange+0x18c
> > amap_wiperange() at amap_pp_adjref+0x28c
> > amap_pp_adjref() at amap_adjref_anons+0xf0
> > amap_adjref_anons() at uvm_unmap_detach+0x70
> > uvm_unmap_detach() at sys_munmap+0x128
> > sys_munmap() at svc_handler+0x478
> > svc_handler() at do_el0_sync+0xc8
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at 0x4f73673c8
> > --- trap ---
> > end of kernel
> > ddb{10}> mach ddbcpu 0xb
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{11}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x59245b4
> > --- trap ---
> > end of kernel
> > ddb{11}> mach ddbcpu 0xc
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{12}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5914b74
> > --- trap ---
> > end of kernel
> > ddb{12}> mach ddbcpu 0xd
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{13}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x4fdc1c0
> > --- trap ---
> > end of kernel
> > ddb{13}> mach ddbcpu 0xe
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{14}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x57038b8
> > --- trap ---
> > end of kernel
> > ddb{14}> mach ddbcpu 0xf
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{15}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5914b1c
> > --- trap ---
> > end of kernel
> > ddb{15}> mach ddbcpu 0x10
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{16}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x4e63188
> > --- trap ---
> > end of kernel
> > ddb{16}> mach ddbcpu 0x11
> > Stopped at __mp_lock+0x138: ldr w8, [x23,#1352]
> > db_enter() at __mp_lock+0x134
> > __mp_lock() at svc_handler+0x42c
> > svc_handler() at do_el0_sync+0xc8
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at 0x46d2d0fe0
> > --- trap ---
> > end of kernel
> > ddb{17}> trace
> > db_enter() at __mp_lock+0x134
> > __mp_lock() at svc_handler+0x42c
> > svc_handler() at do_el0_sync+0xc8
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at 0x46d2d0fe0
> > --- trap ---
> > end of kernel
> > ddb{17}> mach ddbcpu 0x12
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{18}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x5933dd8
> > --- trap ---
> > end of kernel
> > ddb{18}> mach ddbcpu 0x13
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{19}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at 0x4819ddc40
> > --- trap ---
> > end of kernel
> > ddb{19}> mach ddbcpu 0x14
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{20}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x58958ac
> > --- trap ---
> > end of kernel
> > ddb{20}> mach ddbcpu 0x15
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{21}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x4e63188
> > --- trap ---
> > end of kernel
> > ddb{21}> mach ddbcpu 0x16
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{22}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xec
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x594b754
> > --- trap ---
> > end of kernel
> > ddb{22}> mach ddbcpu 0x17
> > Stopped at aplintc_fiq_handler+0x70: b ffffff80006d17c4
> > <aplin
> > tc_fiq_handler+0x7c>
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > ddb{23}> trace
> > db_enter() at aplintc_fiq_handler+0x6c
> > aplintc_fiq_handler() at arm_cpu_fiq+0x34
> > arm_cpu_fiq() at handle_el1h_fiq+0x68
> > handle_el1h_fiq() at mtx_enter+0xe4
> > mtx_enter() at uvm_fault_lower+0x630
> > uvm_fault_lower() at uvm_fault+0x174
> > uvm_fault() at udata_abort+0x138
> > udata_abort() at do_el0_sync+0x130
> > do_el0_sync() at handle_el0_sync+0x70
> > handle_el0_sync() at __ALIGN_SIZE+0x57038b8
> > --- trap ---
> > end of kernel
> > ddb{23}>
> > ddb{23}> show witness /b
> > Number of known direct relationships is 481
> >
> > Lock order reversal between "&mp->mnt_lock"(rwlock) and
> > "&ip->i_lock"(rrwlock)!
> >
> > witness: incomplete path, depth 4
> > ddb{23}> show panic
> > *cpu1: kernel diagnostic assertion "uvm_page_owner_locked_p(pg)" failed:
> > file "
> > /home/dana/src/openbsd/openbsd-src/sys/uvm/uvm_page.c", line 1268
>
--
:wq Claudio
