Since David Gwynne's locking changes in uvm on October 30, I have seen
no more kernel panics on this workload (compiling /usr/src -j24 on
8core Power9) that consistently crashed before. Thank you, sir!
On Tue, Sep 10, 2024 at 3:16 PM Eric Grosse <gro...@gmail.com> wrote:
>
> In case it helps those searching for the race condition in uvm, on
> ppc64 I'm still seeing the problem I reported in May. This currently
> prevents running the Go build server, since I do not know how to
> control parallelism in their LUCI infrastructure. I regret that I
> do not have the skill to narrow this bug report, but am happy to
> test patches.
>
> With a fresh sysupgrade -s, compiling a kernel from source works
> fine and so does "cd /usr/src; make obj && time make build".
> Early in the year, I used to be able to build with -j24 but no longer.
>
> cd /usr/src
> make obj && time make -j8 build
> ...
> cc -O2 -pipe -I/usr/src/lib/csu/powerpc64
> -I/usr/src/lib/csu/../../libexec/ld.so
> -I/usr/src/lib/csu/../../libexec/ld.so/powerpc64 -DBOOT_H="boot.h" -MD
> -MP -c -DRCRT0 -fno-stack-protector -mno-vsx -mno-altivec
> -fno-jump-tables /usr/src/lib/csu/crt0.c -o rcrt0.o
> cc -O2 -pipe -I/usr/src/lib/csu/powerpc64
> -I/usr/src/lib/csu/../../libexec/ld.so
> -I/usr/src/lib/csu/../../libexec/ld.so/powerpc64 -DBOOT_H="boot.h" -MD
> -MP -c /usr/src/lib/csu/crt0.c -o crt0.o
> cc -O2 -pipe -I/usr/src/lib/csu/powerpc64
> -I/usr/src/lib/csu/../../libexec/ld.so
> -I/usr/src/lib/csu/../../libexec/ld.so/powerpc64 -DBOOT_H="boot.h" -MD
> -MP -c -DMCRT0 /usr/src/lib/csu/crt0.c -o gcrt0.o
> cc: error: clang frontend command failed with exit code 139 (use -v to
> see invocation)
> OpenBSD clang version 16.0.6
> Target: powerpc64-unknown-openbsd7.6
> Thread model: posix
> InstalledDir: /usr/bin
> <hangs>
>
> <serial console>
> panic: acquiring blockable sleep lock with spinlock or critical
> section held (rwlock) kmmaplk
>
> Stopped at panic+0x134: ori r0,r0,0x0
> TID PID UID PRFLAGS PFLAGS CPU COMMAND
> 81720 38205 21 0x3 0 0 cc
> *394178 2964 21 0x3 0 1 cc
>
> panic+0x134
> witness_checkorder+0x954
> rw_enter_read+0x8c
>
> vm_map_lock_read_ln+0x38
>
> uvmfault_lookup+0x114
> uvm_fault_check+0x68
> uvm_fault+0x12c
> trap+0x7a4
>
> trapagain+0x4
> --- trap (type 0x300) ---
>
> pmap_enter+0x2a4
>
> uvm_fault_lower_lookup+0x28c
> uvm_fault_lower+0x7c
> uvm_fault+0x200
>
> trap+0x270
>
> trapagain+0x4
>
> --- trap (type 0x400) ---
> End of kernel: 0xbffff175b59e53d0 lr 0x154cc008
>
> ddb{1}> ps
>
> PID TID PPID UID S FLAGS WAIT COMMAND
> 38205 81720 38817 21 7 0x3 cc
> 17131 23183 72654 21 3 0x3 fltagain2 cc
> 38817 342572 61427 21 3 0x83 wait cc
> 85589 76205 80428 21 3 0x3 fltagain2 cc
> 21199 116225 57897 21 3 0x3 fltagain2 cc
>
> 57896 145668 81956 21 3 0x3 fltagain2 cc
> 72654 65372 3668 21 3 0x10008b sigsusp sh
> 61427 342507 3668 21 3 0x10008b sigsusp sh
> * 2964 394178 86705 21 7 0x3 cc
> 80428 109231 3668 21 3 0x10008b sigsusp sh
> 7202 222109 36224 21 3 0x3 fltagain2 cc
> 57897 163653 3668 21 3 0x10008b sigsusp sh
> 81956 255046 3668 21 3 0x10008b sigsusp sh
> 86705 7339 3668 21 3 0x10008b sigsusp sh
> 36224 92799 3668 21 3 0x10008b sigsusp sh
> 3668 477560 95341 21 3 0x10008b sigsusp make
> 95341 399187 51888 21 3 0x10008b sigsusp sh
> 51888 211024 45040 21 3 0x10008b sigsusp make
> 45040 70951 29698 0 3 0x10008b sigsusp sh
>
> 29698 27704 45273 0 3 0x10008b sigsusp make
> 45273 209242 44863 0 3 0x10008b sigsusp make
> 44863 3983 35435 0 3 0x10008b sigsusp sh
> 67504 453328 96712 1000 3 0x100083 ttyin ksh
> 96712 488393 46139 1000 3 0x98 kqread sshd-session
> 46139 468120 75801 0 3 0x92 kqread sshd-session
> 35435 433519 51014 0 3 0x10008b sigsusp sh
> 51014 327045 43152 1000 3 0x10008b sigsusp ksh
> 43152 478898 89809 1000 3 0x98 kqread sshd-session
> 89809 307786 75801 0 3 0x92 kqread sshd-session
>
> 61477 117977 1 0 3 0x100083 ttyin getty
> 86315 419264 1 0 3 0x100098 kqread cron
>
> 75801 8884 1 0 3 0x88 kqread sshd
> 65703 150216 1 0 3 0x100080 kqread ntpd
>
> 9012 328130 61017 83 3 0x100092 kqread ntpd
> 61017 442266 1 83 3 0x1100092 kqread ntpd
> 30689 4244 34908 74 3 0x1100092 bpf pflogd
> 34908 308111 1 0 3 0x80 sbwait pflogd
> 84246 378376 81217 73 3 0x1100090 kqread syslogd
> 81217 138988 1 0 3 0x100082 sbwait syslogd
> 13170 166688 1 0 3 0x100080 kqread resolvd
> 17813 348131 844 77 3 0x100092 kqread dhcpleased
> 32512 508856 844 77 3 0x100092 kqread dhcpleased
> 844 220510 1 0 3 0x80 kqread dhcpleased
> 52994 205415 0 0 3 0x14200 bored smr
> 369 430995 0 0 3 0x14200 pgzero zerothread
> 72295 164419 0 0 3 0x14200 aiodoned aiodoned
> 56284 175686 0 0 3 0x14200 syncer update
> 83703 27996 0 0 3 0x14200 cleaner cleaner
> 11949 408321 0 0 3 0x14200 reaper reaper
> 72532 155655 0 0 3 0x14200 pgdaemon pagedaemon
> 63870 183469 0 0 3 0x14200 usbtsk usbtask
> 29923 22644 0 0 3 0x14200 usbatsk usbatsk
> 57474 363484 0 0 3 0x14200 bored ipmicmd
>
> 57881 470197 0 0 3 0x14200 ipmi_poll ipmi0
> 91327 85282 0 0 3 0x14200 bored sensors
> 22751 335569 0 0 7 0x40014200 idle7
> 47414 145026 0 0 7 0x40014200 idle6
> 80849 11544 0 0 7 0x40014200 idle5
> 97427 34720 0 0 7 0x40014200 idle4
> 4478 37933 0 0 7 0x40014200 idle3
> 73097 335517 0 0 7 0x40014200 idle2
> 99435 318061 0 0 3 0x40014200 idle1
> 95257 384861 0 0 3 0x14200 bored softnet3
> 90330 83959 0 0 3 0x14200 bored softnet2
> 49535 435702 0 0 3 0x14200 bored softnet1
> 82317 423639 0 0 3 0x14200 bored softnet0
> 77438 378027 0 0 3 0x14200 bored systqmp
> 81418 381000 0 0 3 0x14200 bored systq
> 47072 155585 0 0 3 0x14200 tmoslp softclockmp
> 61900 236518 0 0 3 0x40014200 tmoslp softclock
> 11355 448828 0 0 3 0x40014200 idle0
> 37507 277149 0 0 3 0x14200 kmalloc kmthread
> 1 96080 0 0 3 0x82 wait init
> 0 0 -1 0 3 0x10200 scheduler swapper
>
>
> ddb{1}> mach ddbcpu 0
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
>
> xive_hvi+0x1b8
>
> hvi_intr+0x38
> trap+0xd4
>
> trapagain+0x4
>
> --- trap (type 0xea0) ---
>
> uvm_pageactivate+0x128
> uvm_fault_lower+0x558
>
> uvm_fault+0x200
>
> trap+0x4a8
>
> trapagain+0x4
> --- trap (type 0x300) ---
>
> End of kernel: 0xbffff6b310df3f20 lr 0x47b6177c8
>
> ddb{0}> mach ddbcpu 1
>
> Stopped at panic+0x134: ori r0,r0,0x0
> panic+0x134
>
> witness_checkorder+0x954
> rw_enter_read+0x8c
>
> vm_map_lock_read_ln+0x38
> uvmfault_lookup+0x114
>
> uvm_fault_check+0x68
> uvm_fault+0x12c
>
> trap+0x7a4
>
> trapagain+0x4
> --- trap (type 0x300) ---
> pmap_enter+0x2a4
>
> uvm_fault_lower_lookup+0x28c
>
> uvm_fault_lower+0x7c
>
> uvm_fault+0x200
>
> trap+0x270
>
> ddb{1}> mach ddbcpu 2
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
>
> xive_hvi+0x1b8
>
> hvi_intr+0x38
>
> trap+0xd4
> trapagain+0x4
>
> --- trap (type 0xea0) ---
> cpu_idle_cycle+0x88
>
> sched_idle+0x2b8
> proc_trampoline+0x10
>
> ddb{2}> mach ddbcpu 3
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
>
> xive_hvi+0x1b8
> hvi_intr+0x38
>
> trap+0xd4
> trapagain+0x4
>
> --- trap (type 0xea0) ---
> cpu_idle_cycle+0x88
>
> sched_idle+0x2b8
>
> proc_trampoline+0x10
>
> ddb{3}> mach ddbcpu 4
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
> xive_hvi+0x1b8
>
> hvi_intr+0x38
>
> trap+0xd4
>
> trapagain+0x4
>
> --- trap (type 0xea0) ---
>
> cpu_idle_cycle+0x88
> sched_idle+0x2b8
>
> proc_trampoline+0x10
> ddb{4}> mach ddbcpu 5
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
> xive_hvi+0x1b8
>
> hvi_intr+0x38
> trap+0xd4
>
> trapagain+0x4
>
> --- trap (type 0xea0) ---
> cpu_idle_cycle+0x88
>
> sched_idle+0x2b8
>
> proc_trampoline+0x10
>
> ddb{5}> mach ddbcpu 6
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
>
> xive_hvi+0x1b8
> hvi_intr+0x38
>
> trap+0xd4
>
> trapagain+0x4
>
> --- trap (type 0xea0) ---
> cpu_idle_cycle+0x88
>
> sched_idle+0x2b8
>
> proc_trampoline+0x10
>
> ddb{6}> mach ddbcpu 7
>
> Stopped at cpu_intr+0x50: ori r0,r0,0x0
> cpu_intr+0x50
> xive_hvi+0x1b8
>
> hvi_intr+0x38
>
> trap+0xd4
>
> trapagain+0x4
> --- trap (type 0xea0) ---
>
> cpu_idle_cycle+0x88
>
> sched_idle+0x2b8
> proc_trampoline+0x10
>
>
>
> ddb{1}> show uvm
>
> Current UVM status:
> pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
> 63963348 VM pages: 17327 active, 51113 inactive, 1 wired, 63701662 free
> (7962
> 771 zero)
> min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
> freemin=2132111, free-target=2842814, inactive-target=0, wired-max=21321116
> faults=6174613, traps=6705939, intrs=1463968, ctxswitch=1099578 fpuswitch=0
>
> softint=98673, syscalls=3851749, kmapent=11
> fault counts:
> noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
>
> ok relocks(total)=82734(85565), anget(retries)=2875306(0),
> amapcopy=1057027
>
> neighbor anon/obj pg=1748628/4763470, gets(lock/unlock)=1572156/85600
> cases: anon=2304774, anoncow=570532, obj=1323548, prcopy=245741,
> przero=173
> 0010
> daemon and swap counts:
>
> woke=0, revs=0, scans=0, obscans=0, anscans=0
> busy=0, freed=0, reactivate=0, deactivate=0
> pageouts=0, pending=0, nswget=0
>
> nswapdev=1
>
> swpages=11948184, swpginuse=0, swpgonly=0 paging=0
> kernel pointers:
> objs(kern)=0x1406378
>
> ddb{1}> show bcstats
>
> Current Buffer Cache status:
> numbufs 42996 busymapped 0, delwri 3147
> kvaslots 52428 avail kva slots 52428
> bufpages 171980, dmapages 171980, dirtypages 12588
> pendingreads 7, pendingwrites 0
> highflips 0, highflops 0, dmaflips 0
>
>
> <dmesg>
> OpenBSD 7.6-beta (GENERIC.MP) #31: Tue Sep 10 18:25:36 UTC 2024
> e...@t2.lan:/sys/arch/powerpc64/compile/GENERIC.MP
> real mem = 274877906944 (262144MB)
> avail mem = 261989404672 (249852MB)
> random: good seed from bootblocks
> mainbus0 at root: T2P9S01 REV 1.01
> cpu0 at mainbus0 pir 4: IBM POWER9 2.3, 2700 MHz
> cpu0: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu0: 512KB 128b/line 8-way L2 cache
> cpu0: 10MB 128b/line 8-way L3 cache
> cpu1 at mainbus0 pir c: IBM POWER9 2.3, 2700 MHz
> cpu1: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu1: 512KB 128b/line 8-way L2 cache
> cpu1: 10MB 128b/line 8-way L3 cache
> cpu2 at mainbus0 pir 14: IBM POWER9 2.3, 2700 MHz
> cpu2: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu2: 512KB 128b/line 8-way L2 cache
> cpu2: 10MB 128b/line 8-way L3 cache
> cpu3 at mainbus0 pir 1c: IBM POWER9 2.3, 2700 MHz
> cpu3: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu3: 512KB 128b/line 8-way L2 cache
> cpu3: 10MB 128b/line 8-way L3 cache
> cpu4 at mainbus0 pir 804: IBM POWER9 2.3, 2700 MHz
> cpu4: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu4: 512KB 128b/line 8-way L2 cache
> cpu4: 10MB 128b/line 8-way L3 cache
> cpu5 at mainbus0 pir 80c: IBM POWER9 2.3, 2700 MHz
> cpu5: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu5: 512KB 128b/line 8-way L2 cache
> cpu5: 10MB 128b/line 8-way L3 cache
> cpu6 at mainbus0 pir 814: IBM POWER9 2.3, 2700 MHz
> cpu6: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu6: 512KB 128b/line 8-way L2 cache
> cpu6: 10MB 128b/line 8-way L3 cache
> cpu7 at mainbus0 pir 81c: IBM POWER9 2.3, 2700 MHz
> cpu7: 32KB 128b/line 8-way L1 I-cache, 32KB 128b/line 8-way L1 D-cache
> cpu7: 512KB 128b/line 8-way L2 cache
> cpu7: 10MB 128b/line 8-way L3 cache
> "bmc" at mainbus0 not configured
> "ibm,firmware-versions" at mainbus0 not configured
> "ibm,hostboot" at mainbus0 not configured
> opal0 at mainbus0: skiboot-9858186
> opal0: idle psscr 300332
> opalcons0 at opal0
> opalsens0 at opal0: "core-temp"
> opalsens1 at opal0: "core-temp"
> opalsens2 at opal0: "core-temp"
> opalsens3 at opal0: "core-temp"
> opalsens4 at opal0: "core-temp"
> opalsens5 at opal0: "core-temp"
> opalsens6 at opal0: "core-temp"
> opalsens7 at opal0: "core-temp"
> opalsens8 at opal0: "mem-temp"
> opalsens9 at opal0: "mem-temp"
> opalsens10 at opal0: "mem-temp"
> opalsens11 at opal0: "mem-temp"
> opalsens12 at opal0: "mem-temp"
> opalsens13 at opal0: "mem-temp"
> opalsens14 at opal0: "mem-temp"
> opalsens15 at opal0: "mem-temp"
> opalsens16 at opal0: "mem-temp"
> opalsens17 at opal0: "mem-temp"
> opalsens18 at opal0: "mem-temp"
> opalsens19 at opal0: "mem-temp"
> opalsens20 at opal0: "mem-temp"
> opalsens21 at opal0: "mem-temp"
> opalsens22 at opal0: "mem-temp"
> opalsens23 at opal0: "mem-temp"
> opalsens24 at opal0: "mem-temp"
> opalsens25 at opal0: "mem-temp"
> opalsens26 at opal0: "mem-temp"
> opalsens27 at opal0: "mem-temp"
> opalsens28 at opal0: "mem-temp"
> opalsens29 at opal0: "mem-temp"
> opalsens30 at opal0: "mem-temp"
> opalsens31 at opal0: "mem-temp"
> opalsens32 at opal0: "mem-temp"
> opalsens33 at opal0: "mem-temp"
> opalsens34 at opal0: "mem-temp"
> opalsens35 at opal0: "mem-temp"
> opalsens36 at opal0: "mem-temp"
> opalsens37 at opal0: "mem-temp"
> opalsens38 at opal0: "mem-temp"
> opalsens39 at opal0: "mem-temp"
> opalsens40 at opal0: "proc-energy"
> opalsens41 at opal0: "proc-energy"
> opalsens42 at opal0: "proc-energy"
> opalsens43 at opal0: "proc-energy"
> opalsens44 at opal0: "proc-energy"
> opalsens45 at opal0: "proc-energy"
> opalsens46 at opal0: "proc-in"
> opalsens47 at opal0: "proc-in"
> opalsens48 at opal0: "proc-in"
> opalsens49 at opal0: "proc-in"
> opalsens50 at opal0: "proc-power"
> opalsens51 at opal0: "proc-power"
> opalsens52 at opal0: "proc-power"
> opalsens53 at opal0: "proc-power"
> opalsens54 at opal0: "proc-power"
> opalsens55 at opal0: "proc-power"
> opalsens56 at opal0: "proc-temp"
> opalsens57 at opal0: "proc-temp"
> opalsens58 at opal0: "vrm-curr"
> opalsens59 at opal0: "vrm-curr"
> opalsens60 at opal0: "vrm-curr"
> opalsens61 at opal0: "vrm-curr"
> opalsens62 at opal0: "vrm-in"
> opalsens63 at opal0: "vrm-in"
> opalsens64 at opal0: "vrm-in"
> opalsens65 at opal0: "vrm-in"
> opalsens66 at opal0: "vrm-temp"
> opalsens67 at opal0: "vrm-temp"
> ipmi0 at opal0: version 2.0 interface OPAL
> "ibm,pcie-slots" at mainbus0 not configured
> "ibm,secureboot" at mainbus0 not configured
> "imc-counters" at mainbus0 not configured
> xics0 at mainbus0
> xive0 at mainbus0
> "ipl-params" at mainbus0 not configured
> "lpcm-opb" at mainbus0 not configured
> phb0 at mainbus0: chip 0x0
> pci0 at phb0
> ppb0 at pci0 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci1 at ppb0 bus 1
> em0 at pci1 dev 0 function 0 "Intel 82576" rev 0x01: msi, address
> 80:61:5f:0e:89:92
> em1 at pci1 dev 0 function 1 "Intel 82576" rev 0x01: msi, address
> 80:61:5f:0e:89:93
> phb1 at mainbus0: chip 0x0
> pci2 at phb1
> ppb1 at pci2 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci3 at ppb1 bus 1
> nvme0 at pci3 dev 0 function 0 "Samsung SM961/PM961 NVMe" rev 0x00:
> msix, NVMe 1.2
> nvme0: Samsung SSD 960 EVO 500GB, firmware 3B7QCXE7, serial S3X4NB0K309827P
> scsibus0 at nvme0: 2 targets, initiator 0
> sd0 at scsibus0 targ 1 lun 0: <NVMe, Samsung SSD 960, 3B7Q>
> sd0: 476940MB, 512 bytes/sector, 976773168 sectors
> phb2 at mainbus0: chip 0x0
> pci4 at phb2
> ppb2 at pci4 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci5 at ppb2 bus 1
> phb3 at mainbus0: chip 0x0
> pci6 at phb3
> ppb3 at pci6 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci7 at ppb3 bus 1
> xhci0 at pci7 dev 0 function 0 "TI xHCI" rev 0x02: msix, xHCI 0.96
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "TI xHCI root hub" rev
> 3.00/1.00 addr 1
> phb4 at mainbus0: chip 0x0
> pci8 at phb4
> ppb4 at pci8 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci9 at ppb4 bus 1
> bge0 at pci9 dev 0 function 0 "Broadcom BCM5719" rev 0x00, BCM5719 A1
> (0x5719001), APE firmware NCSI 0.6.0.18: msi, address
> 2c:09:4d:00:07:db
> brgphy0 at bge0 phy 1: BCM5719C, rev. 0
> bge1 at pci9 dev 0 function 1 "Broadcom BCM5719" rev 0x00, BCM5719 A1
> (0x5719001), APE firmware NCSI 0.6.0.18: msi, address
> 2c:09:4d:00:07:dc
> brgphy1 at bge1 phy 2: BCM5719C, rev. 0
> phb5 at mainbus0: chip 0x0
> pci10 at phb5
> ppb5 at pci10 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci11 at ppb5 bus 1
> ppb6 at pci11 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x04
> pci12 at ppb6 bus 2
> astfb0 at pci12 dev 0 function 0 "ASPEED Technology AST2000" rev 0x41:
> can't map framebuffer
> phb6 at mainbus0: chip 0x8
> pci13 at phb6
> ppb7 at pci13 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci14 at ppb7 bus 1
> phb7 at mainbus0: chip 0x8
> pci15 at phb7
> ppb8 at pci15 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci16 at ppb8 bus 1
> phb8 at mainbus0: chip 0x8
> pci17 at phb8
> ppb9 at pci17 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci18 at ppb9 bus 1
> phb9 at mainbus0: chip 0x8
> pci19 at phb9
> ppb10 at pci19 dev 0 function 0 "IBM POWER9 Host" rev 0x00
> pci20 at ppb10 bus 1
> "psi" at mainbus0 not configured
> "psi" at mainbus0 not configured
> "vas" at mainbus0 not configured
> "vas" at mainbus0 not configured
> "vpd" at mainbus0 not configured
> "xscom" at mainbus0 not configured
> "xscom" at mainbus0 not configured
> uhidev0 at uhub0 port 2 configuration 1 interface 0 "EATON Eaton 5SC"
> rev 1.10/1.00 addr 2
> uhidev0: iclass 3/0, 255 report ids
> upd0 at uhidev0
> uhid0 at uhidev0 reportid 3: input=1, output=0, feature=1
> uhid1 at uhidev0 reportid 5: input=1, output=0, feature=1
> uhid2 at uhidev0 reportid 6: input=4, output=0, feature=4
> uhid3 at uhidev0 reportid 7: input=0, output=0, feature=2
> uhid4 at uhidev0 reportid 8: input=0, output=0, feature=1
> uhid5 at uhidev0 reportid 9: input=0, output=0, feature=4
> uhid6 at uhidev0 reportid 10: input=0, output=0, feature=4
> uhid7 at uhidev0 reportid 11: input=0, output=0, feature=5
> uhid8 at uhidev0 reportid 12: input=0, output=0, feature=1
> uhid9 at uhidev0 reportid 13: input=0, output=0, feature=2
> uhid10 at uhidev0 reportid 14: input=0, output=0, feature=1
> uhid11 at uhidev0 reportid 15: input=1, output=0, feature=1
> uhid12 at uhidev0 reportid 16: input=0, output=0, feature=7
> uhid13 at uhidev0 reportid 17: input=0, output=0, feature=1
> uhid14 at uhidev0 reportid 18: input=0, output=0, feature=1
> uhid15 at uhidev0 reportid 19: input=0, output=0, feature=1
> uhid16 at uhidev0 reportid 20: input=0, output=0, feature=1
> uhid17 at uhidev0 reportid 21: input=0, output=0, feature=2
> uhid18 at uhidev0 reportid 22: input=0, output=0, feature=1
> uhid19 at uhidev0 reportid 24: input=0, output=0, feature=4
> uhid20 at uhidev0 reportid 32: input=0, output=0, feature=3
> uhid21 at uhidev0 reportid 34: input=0, output=0, feature=1
> uhid22 at uhidev0 reportid 35: input=0, output=0, feature=4
> uhid23 at uhidev0 reportid 36: input=0, output=0, feature=1
> uhid24 at uhidev0 reportid 37: input=0, output=0, feature=4
> uhid25 at uhidev0 reportid 38: input=0, output=0, feature=2
> uhid26 at uhidev0 reportid 39: input=6, output=0, feature=6
> uhid27 at uhidev0 reportid 40: input=0, output=0, feature=1
> uhid28 at uhidev0 reportid 41: input=0, output=0, feature=1
> uhid29 at uhidev0 reportid 48: input=0, output=0, feature=2
> uhid30 at uhidev0 reportid 49: input=0, output=0, feature=2
> uhid31 at uhidev0 reportid 50: input=6, output=0, feature=6
> uhid32 at uhidev0 reportid 57: input=0, output=0, feature=1
> uhid33 at uhidev0 reportid 58: input=0, output=0, feature=1
> uhid34 at uhidev0 reportid 59: input=1, output=0, feature=1
> uhid35 at uhidev0 reportid 60: input=0, output=0, feature=1
> uhid36 at uhidev0 reportid 61: input=0, output=0, feature=2
> uhid37 at uhidev0 reportid 62: input=0, output=0, feature=1
> uhid38 at uhidev0 reportid 64: input=0, output=0, feature=1
> uhid39 at uhidev0 reportid 65: input=1, output=0, feature=1
> uhid40 at uhidev0 reportid 66: input=0, output=0, feature=2
> uhid41 at uhidev0 reportid 67: input=0, output=0, feature=1
> uhid42 at uhidev0 reportid 71: input=0, output=0, feature=2
> uhid43 at uhidev0 reportid 72: input=0, output=0, feature=2
> uhid44 at uhidev0 reportid 73: input=1, output=0, feature=1
> uhid45 at uhidev0 reportid 74: input=0, output=0, feature=1
> uhid46 at uhidev0 reportid 77: input=0, output=0, feature=2
> uhid47 at uhidev0 reportid 78: input=0, output=0, feature=2
> uhid48 at uhidev0 reportid 79: input=0, output=0, feature=1
> uhid49 at uhidev0 reportid 113: input=0, output=0, feature=30
> uhid50 at uhidev0 reportid 114: input=0, output=0, feature=2
> uhid51 at uhidev0 reportid 115: input=0, output=0, feature=1
> uhid52 at uhidev0 reportid 116: input=0, output=0, feature=2
> uhid53 at uhidev0 reportid 117: input=0, output=0, feature=2
> uhid54 at uhidev0 reportid 128: input=0, output=0, feature=1
> uhid55 at uhidev0 reportid 129: input=0, output=0, feature=1
> uhid56 at uhidev0 reportid 253: input=0, output=0, feature=1
> uhid57 at uhidev0 reportid 254: input=0, output=0, feature=10
> uhid58 at uhidev0 reportid 255: input=0, output=0, feature=63
> uhub1 at uhub0 port 3 configuration 1 interface 0 "Aspeed USB Virtual
> Hub" rev 2.00/1.00 addr 3
> uhidev1 at uhub1 port 1 configuration 1 interface 0 "OpenBMC
> virtual_input" rev 2.00/1.00 addr 4
> uhidev1: iclass 3/1
> ukbd0 at uhidev1: 8 variable keys, 6 key codes
> wskbd0 at ukbd0 mux 1
> uhidev2 at uhub1 port 1 configuration 1 interface 1 "OpenBMC
> virtual_input" rev 2.00/1.00 addr 4
> uhidev2: iclass 3/1
> ums0 at uhidev2: 3 buttons
> wsmouse0 at ums0 mux 0
> uhub2 at uhub0 port 4 configuration 1 interface 0 "Genesys Logic
> USB2.0 Hub" rev 2.00/32.98 addr 5
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
> root on sd0a (f115e2b2a7d7c47d.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> uhub1: illegal enable change, port 1
