On Sat, Nov 20, 2021 at 02:45:15PM +0100, Christopher Zimmermann wrote:
On Sat, Nov 20, 2021 at 02:31:37PM +0100, Mark Kettenis wrote:
Date: Fri, 19 Nov 2021 21:56:19 +0100 (CET)
From: Mark Kettenis <mark.kette...@xs4all.nl>
Date: Fri, 19 Nov 2021 21:36:55 +0100
From: Christopher Zimmermann <chr...@openbsd.org>
It is about MSI. After disabling MSI interrupts are delivered, albeit
the performance seems to be a bit poor (reading at a constant 8 MB/s on
a WD Red disk).
Here's the dmesg with patched kernel - notice the intx instead msi:
ahci0 at pci1 dev 0 function 0 "JMicron JMB58x AHCI" rev 0x00: intx, AHCI 1.3.1
Should I commit this quirk? Or might this break something on other
platforms?
Before we do this, we should establish whether this is a rkpcie(4)
problem or if that PCI device really has an issue with MSIs.
So MSIs still work fine on my firefly-rk3399:
nvme0 at pci1 dev 0 function 0 "Silicon Motion SM2260 NVMe" rev 0x03: msix,
NVMe 1.2
nvme0: ADATA SX8000NP, firmware C2.2.1, serial 2H1220011170
scsibus0 at nvme0: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <NVMe, ADATA SX8000NP, C2.2>
sd0: 122104MB, 512 bytes/sector, 250069680 sectors
Can you post a full dmesg for your RockPro64, and a pcidump -vxxx
output of possible?
sure.
Ah, sorry, you need a dmesg and pcidump from an unpatched kernel I
suspect. I cannot boot a vanilla kernel right now, so cannot get you a
vanilla pcidump, but here's a dmesg from yesterday with ahci, xhci and
uhub debugging enabled:
OpenBSD 7.0-current (GENERIC.MP) #10: Fri Nov 19 06:24:32 CET 2021
madro...@wanja.gmerlin.de:/usr/src/sys/arch/arm64/compile/GENERIC.MP
real mem = 4038799360 (3851MB)
avail mem = 3839606784 (3661MB)
random: good seed from bootblocks
mainbus0 at root: Pine64 RockPro64 v2.1
psci0 at mainbus0: PSCI 1.1, SMCCC 1.2
cpu0 at mainbus0 mpidr 0: ARM Cortex-A53 r0p4
cpu0: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu1 at mainbus0 mpidr 1: ARM Cortex-A53 r0p4
cpu1: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu1: 512KB 64b/line 16-way L2 cache
cpu1: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu2 at mainbus0 mpidr 2: ARM Cortex-A53 r0p4
cpu2: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu2: 512KB 64b/line 16-way L2 cache
cpu2: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu3 at mainbus0 mpidr 3: ARM Cortex-A53 r0p4
cpu3: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu3: 512KB 64b/line 16-way L2 cache
cpu3: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu4 at mainbus0 mpidr 100: ARM Cortex-A72 r0p2
cpu4: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
cpu4: 1024KB 64b/line 16-way L2 cache
cpu4: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu5 at mainbus0 mpidr 101: ARM Cortex-A72 r0p2
cpu5: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
cpu5: 1024KB 64b/line 16-way L2 cache
cpu5: CRC32,SHA2,SHA1,AES+PMULL,ASID16
efi0 at mainbus0: UEFI 2.8
efi0: Das U-Boot rev 0x20211000
apm0 at mainbus0
agintc0 at mainbus0 sec shift 3:3 nirq 288 nredist 6 ipi: 0, 1:
"interrupt-controller"
agintcmsi0 at agintc0
syscon0 at mainbus0: "qos"
syscon1 at mainbus0: "qos"
syscon2 at mainbus0: "qos"
syscon3 at mainbus0: "qos"
syscon4 at mainbus0: "qos"
syscon5 at mainbus0: "qos"
syscon6 at mainbus0: "qos"
syscon7 at mainbus0: "qos"
syscon8 at mainbus0: "qos"
syscon9 at mainbus0: "qos"
syscon10 at mainbus0: "qos"
syscon11 at mainbus0: "qos"
syscon12 at mainbus0: "qos"
syscon13 at mainbus0: "qos"
syscon14 at mainbus0: "qos"
syscon15 at mainbus0: "qos"
syscon16 at mainbus0: "qos"
syscon17 at mainbus0: "qos"
syscon18 at mainbus0: "qos"
syscon19 at mainbus0: "qos"
syscon20 at mainbus0: "qos"
syscon21 at mainbus0: "qos"
syscon22 at mainbus0: "qos"
syscon23 at mainbus0: "qos"
syscon24 at mainbus0: "qos"
syscon25 at mainbus0: "power-management"
"power-controller" at syscon25 not configured
syscon26 at mainbus0: "syscon"
"io-domains" at syscon26 not configured
rkclock0 at mainbus0
rkclock1 at mainbus0
syscon27 at mainbus0: "syscon"
"io-domains" at syscon27 not configured
"usb2phy" at syscon27 not configured
"usb2phy" at syscon27 not configured
rkemmcphy0 at syscon27
"pcie-phy" at syscon27 not configured
rktcphy0 at mainbus0
rktcphy1 at mainbus0
rkpinctrl0 at mainbus0: "pinctrl"
rkgpio0 at rkpinctrl0
rkgpio1 at rkpinctrl0
rkgpio2 at rkpinctrl0
rkgpio3 at rkpinctrl0
rkgpio4 at rkpinctrl0
pwmreg0 at mainbus0
rkdrm0 at mainbus0
drm0 at rkdrm0
"pmu_a53" at mainbus0 not configured
"pmu_a72" at mainbus0 not configured
agtimer0 at mainbus0: 24000 kHz
"xin24m" at mainbus0 not configured
rkpcie0 at mainbus0
pci0 at rkpcie0
ppb0 at pci0 dev 0 function 0 "Rockchip RK3399 Root Complex" rev 0x00: msi
pci1 at ppb0 bus 1
ahci0 at pci1 dev 0 function 0 vendor "JMicron", unknown product 0x0585 rev
0x00: msi, GHC 0x0 AHCI 1.3.1
ahci0: capabilities
0xef33ff84<S64A,NCQ,SSNTF,SSS,SALP,SAL,SCLO,SPM,FBSS,PMD,SSC,PSC,CCCS>, 5
ports, 32 cmds, gen 3 (6.0Gb/s)
ahci0: extended capabilities 0x5<APST,BOH>
ahci0: ports implemented: 0x00000003
ahci0.0: port reset
ahci0.1: port reset
ahci0.0: PMP probe 2
ahci0.0: sending PMP reset cmd
ahci0: poll
ahci0.0: slot 1 is complete
ahci0.0: sending PMP probe status cmd
ahci0: poll
ahci0.0: PMP probe cmd timed out
ahci0.0: timed out 1, active 0, active_cnt 0
ahci0.0: PMP probe status cmd failed
ahci0.0: PMP probe 1
ahci0.0: sending PMP reset cmd
ahci0: poll
ahci0.0: slot 1 is complete
ahci0.0: sending PMP probe status cmd
ahci0: poll
ahci0.0: PMP probe cmd timed out
ahci0.0: timed out 1, active 0, active_cnt 0
ahci0.0: PMP probe status cmd failed
ahci0.0: no PMP found, resetting the port
ahci0: detected device on port 0; 0
ahci0.0: port 0: 6.0Gb/s
ahci0: no device detected on port 1
scsibus0 at ahci0: 32 targets
ahci0: poll
ahci0.0: port interrupt: 22<DPS,PSS>
ahci0.0: slot 0 is complete
ahci0: poll
ahci0.0: port interrupt: 1<DHRS>
ahci0.0: slot 2 is complete
ahci0: poll
ahci0.0: port interrupt: 1<DHRS>
ahci0.0: slot 3 is complete
ahci0: poll
ahci0.0: port interrupt: 1<DHRS>
ahci0.0: slot 4 is complete
ahci0: poll
ahci0.0: port interrupt: 1<DHRS>
ahci0.0: slot 5 is complete
sd0 at scsibus0 targ 0 lun 0: <ATA, WDC WD10EFRX-68P, 01.0> naa.50014ee2b4f41c4a
sd0: 953869MB, 512 bytes/sector, 1953525168 sectors
dwge0 at mainbus0: rev 0x35, address 7e:7f:30:d2:3d:e0
rgephy0 at dwge0 phy 0: RTL8169S/8110S/8211 PHY, rev. 6
dwmmc0 at mainbus0: 50 MHz base clock
sdmmc0 at dwmmc0: 4-bit, sd high-speed, dma
dwmmc1 at mainbus0: 50 MHz base clock
sdmmc1 at dwmmc1: 4-bit, sd high-speed, dma
sdhc0 at mainbus0
sdhc0: SDHC 3.0, 200 MHz base clock
sdmmc2 at sdhc0: 8-bit, sd high-speed, mmc high-speed, dma
ehci0 at mainbus0
usb0 at ehci0: USB revision 2.0
usbd_new_device: bus=0xffffff800b5ed000 port=0 depth=0 speed=3
usbd_setup_pipe: dev=0xffffff800b5d5800 iface=0x0 ep=0xffffff800b5d5838
pipe=0xffffff800b5d5808
usbd_new_device: adding unit addr=1, rev=200, class=9, subclass=0, protocol=1,
maxpacket=64, len=18, speed=3
usbd_new_device: new dev (addr 1), dev=0xffffff800b5d5800,
parent=0xffffff800b5d5700
usbd_probe_and_attach trying device specific drivers
usbd_probe_and_attach: no device specific driver found
usbd_probe_and_attach: looping over 1 configurations
usbd_set_config_index: status=0x0001, error=NORMAL_COMPLETION
usbd_set_config_index: (addr 1) cno=1 attr=0xc0, selfpowered=1, power=0
usbd_set_config_index: set config 1
uhub0 at usb0 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00
addr 1
uhub0: 1 port with 1 removable, self powered
usbd_setup_pipe: dev=0xffffff800b5d5800 iface=0xffffff800b5c7340
ep=0xffffff800b596b80 pipe=0xffffff8000f1e7b8
ohci0 at mainbus0: version 1.0
ehci1 at mainbus0
usb1 at ehci1: USB revision 2.0
usbd_new_device: bus=0xffffff800b5f3000 port=0 depth=0 speed=3
usbd_setup_pipe: dev=0xffffff800b5d5a00 iface=0x0 ep=0xffffff800b5d5a38
pipe=0xffffff800b5d5a08
usbd_new_device: adding unit addr=1, rev=200, class=9, subclass=0, protocol=1,
maxpacket=64, len=18, speed=3
usbd_new_device: new dev (addr 1), dev=0xffffff800b5d5a00,
parent=0xffffff800b5d5900
usbd_probe_and_attach trying device specific drivers
usbd_probe_and_attach: no device specific driver found
usbd_probe_and_attach: looping over 1 configurations
usbd_set_config_index: status=0x0001, error=NORMAL_COMPLETION
usbd_set_config_index: (addr 1) cno=1 attr=0xc0, selfpowered=1, power=0
usbd_set_config_index: set config 1
uhub1 at usb1 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00
addr 1
uhub1: 1 port with 1 removable, self powered
usbd_setup_pipe: dev=0xffffff800b5d5a00 iface=0xffffff800b5c7440
ep=0xffffff800b596cc0 pipe=0xffffff8000f1e7b8
ohci1 at mainbus0: version 1.0
rkdwusb0 at mainbus0: "usb"
xhci0 at rkdwusb0, xHCI 1.10
xhci0: CAPLENGTH=0x20
xhci0: DOORBELL=0x480
xhci0: RUNTIME=0x440
xhci0: 64 bytes context
xhci0: supported page size 0x00000001
xhci0: 2 ports and 64 slots
xhci0: 1 scratch pages, ETE=0, IST=0x1
usb2 at xhci0: USB revision 3.0
usbd_new_device: bus=0xffffff800b5f9000 port=0 depth=0 speed=4
usbd_setup_pipe: dev=0xffffff800b5d5c00 iface=0x0 ep=0xffffff800b5d5c38
pipe=0xffffff800b5d5c08
usbd_new_device: adding unit addr=1, rev=300, class=9, subclass=0, protocol=1,
maxpacket=9, len=18, speed=4
usbd_new_device: new dev (addr 1), dev=0xffffff800b5d5c00,
parent=0xffffff800b5d5b00
usbd_probe_and_attach trying device specific drivers
usbd_probe_and_attach: no device specific driver found
usbd_probe_and_attach: looping over 1 configurations
usbd_set_config_index: status=0x0001, error=NORMAL_COMPLETION
usbd_set_config_index: (addr 1) cno=1 attr=0xc0, selfpowered=1, power=0
usbd_set_config_index: set config 1
uhub2 at usb2 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00
addr 1
uhub2: 2 ports with 2 removable, self powered
usbd_setup_pipe: dev=0xffffff800b5d5c00 iface=0xffffff800b5c7580
ep=0xffffff800b596e70 pipe=0xffffff8000f1e618
xhci0: DCBAAP=00x2e7f000
xhci0: CRCR=00 (0000000002e80000)
xhci0: ERSTBA=00xef0c8000
xhci0: ERDP=00x2e81000
xhci0: USBCMD=0x5
xhci0: IMAN=0x2
rkdwusb1 at mainbus0: "usb"
xhci1 at rkdwusb1, xHCI 1.10
xhci1: CAPLENGTH=0x20
xhci1: DOORBELL=0x480
xhci1: RUNTIME=0x440
xhci1: 64 bytes context
xhci1: supported page size 0x00000001
xhci1: 2 ports and 64 slots
xhci1: 1 scratch pages, ETE=0, IST=0x1
usb3 at xhci1: USB revision 3.0
usbd_new_device: bus=0xffffff800b612000 port=0 depth=0 speed=4
usbd_setup_pipe: dev=0xffffff800b5d5e00 iface=0x0 ep=0xffffff800b5d5e38
pipe=0xffffff800b5d5e08
usbd_new_device: adding unit addr=1, rev=300, class=9, subclass=0, protocol=1,
maxpacket=9, len=18, speed=4
usbd_new_device: new dev (addr 1), dev=0xffffff800b5d5e00,
parent=0xffffff800b5d5d00
usbd_probe_and_attach trying device specific drivers
usbd_probe_and_attach: no device specific driver found
usbd_probe_and_attach: looping over 1 configurations
usbd_set_config_index: status=0x0001, error=NORMAL_COMPLETION
usbd_set_config_index: (addr 1) cno=1 attr=0xc0, selfpowered=1, power=0
usbd_set_config_index: set config 1
uhub3 at usb3 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00
addr 1
uhub3: 2 ports with 2 removable, self powered
usbd_setup_pipe: dev=0xffffff800b5d5e00 iface=0xffffff800b5c7680
ep=0xffffff800b596fa0 pipe=0xffffff8000f1e618
xhci1: DCBAAP=00x2e83000
xhci1: CRCR=00 (0000000002e84000)
xhci1: ERSTBA=00xef0c6000
xhci1: ERDP=00x2e85000
xhci1: USBCMD=0x5
xhci1: IMAN=0x2
"saradc" at mainbus0 not configured
rkiic0 at mainbus0
iic0 at rkiic0
escodec0 at iic0 addr 0x11
rkiic1 at mainbus0
iic1 at rkiic1
com0 at mainbus0: ns16550, no working fifo
com1 at mainbus0: ns16550, no working fifo
com1: console
"spi" at mainbus0 not configured
rktemp0 at mainbus0
rkrng0 at mainbus0
rkiic2 at mainbus0
iic2 at rkiic2
rkpmic0 at iic2 addr 0x1b: RK808
fanpwr0 at iic2 addr 0x40: SYR827, 1.20 VDC
fanpwr1 at iic2 addr 0x41: SYR828, 1.00 VDC
rkiic3 at mainbus0
iic3 at rkiic3
fusbtc0 at iic3 addr 0x22
rkpwm0 at mainbus0
rkpwm1 at mainbus0
rkpwm2 at mainbus0
"video-codec" at mainbus0 not configured
"iommu" at mainbus0 not configured
"video-codec" at mainbus0 not configured
"iommu" at mainbus0 not configured
"rga" at mainbus0 not configured
"efuse" at mainbus0 not configured
"dma-controller" at mainbus0 not configured
"dma-controller" at mainbus0 not configured
"watchdog" at mainbus0 not configured
"rktimer" at mainbus0 not configured
rkiis0 at mainbus0
rkiis1 at mainbus0
rkiis2 at mainbus0
rkvop0 at mainbus0: RK3399 VOPL
"iommu" at mainbus0 not configured
rkvop1 at mainbus0: RK3399 VOPB
"iommu" at mainbus0 not configured
essage repeated 2 times
simpleaudio0 at mainbus0
rkdwhdmi0 at mainbus0: HDMI TX
rkdwhdmi0: version 2.11a, phytype 0xf3
"gpu" at mainbus0 not configured
"opp-table0" at mainbus0 not configured
"opp-table1" at mainbus0 not configured
"opp-table2" at mainbus0 not configured
"external-gmac-clock" at mainbus0 not configured
gpiokeys0 at mainbus0: "GPIO Key Power"
"ir-receiver" at mainbus0 not configured
gpioleds0 at mainbus0: "work", "diy"
pwmfan0 at mainbus0: no cooling levels
"sdio-pwrseq" at mainbus0 not configured
graphaudio0 at mainbus0
graphaudio1 at mainbus0
"spdif-dit" at mainbus0 not configured
"vcc12v-dcin" at mainbus0 not configured
"vcc1v8-s3" at mainbus0 not configured
"vcc3v0-sd" at mainbus0 not configured
"vcc3v3-pcie-regulator" at mainbus0 not configured
"vcc3v3-sys" at mainbus0 not configured
"vcc5v0-host-regulator" at mainbus0 not configured
"vcc5v0-typec-regulator" at mainbus0 not configured
"vcc5v0-sys" at mainbus0 not configured
"vcc5v0-usb" at mainbus0 not configured
usb4 at ohci0: USB revision 1.0
usbd_new_device: bus=0xffffff800b5f1000 port=0 depth=0 speed=2
usbd_setup_pipe: dev=0xffffff800b63df00 iface=0x0 ep=0xffffff800b63df38
pipe=0xffffff800b63df08
usbd_new_device: adding unit addr=1, rev=100, class=9, subclass=0, protocol=0,
maxpacket=64, len=18, speed=2
usbd_new_device: new dev (addr 1), dev=0xffffff800b63df00,
parent=0xffffff800b63de00
usbd_probe_and_attach trying device specific drivers
usbd_probe_and_attach: no device specific driver found
usbd_probe_and_attach: looping over 1 configurations
usbd_set_config_index: status=0x0001, error=NORMAL_COMPLETION
usbd_set_config_index: (addr 1) cno=1 attr=0xc0, selfpowered=1, power=0
usbd_set_config_index: set config 1
uhub4 at usb4 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00
addr 1
uhub4: 1 port with 1 removable, self powered
usbd_setup_pipe: dev=0xffffff800b63df00 iface=0xffffff800b6420c0
ep=0xffffff800b596950 pipe=0xffffff8000f1e968
usb5 at ohci1: USB revision 1.0
usbd_new_device: bus=0xffffff800b5f7000 port=0 depth=0 speed=2
usbd_setup_pipe: dev=0xffffff800b643000 iface=0x0 ep=0xffffff800b643038
pipe=0xffffff800b643008
usbd_new_device: adding unit addr=1, rev=100, class=9, subclass=0, protocol=0,
maxpacket=64, len=18, speed=2
usbd_new_device: new dev (addr 1), dev=0xffffff800b643000,
parent=0xffffff800b63db00
usbd_probe_and_attach trying device specific drivers
usbd_probe_and_attach: no device specific driver found
usbd_probe_and_attach: looping over 1 configurations
usbd_set_config_index: status=0x0001, error=NORMAL_COMPLETION
usbd_set_config_index: (addr 1) cno=1 attr=0xc0, selfpowered=1, power=0
usbd_set_config_index: set config 1
uhub5 at usb5 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00
addr 1
uhub5: 1 port with 1 removable, self powered
usbd_setup_pipe: dev=0xffffff800b643000 iface=0xffffff800b642200
ep=0xffffff800b596920 pipe=0xffffff8000f1e968
audio0 at graphaudio0
dt: 445 probes
usb_xfer_abort_thread: start
usb_task_thread: start
uhub0: port 1 status=0x0500 change=0x0000
usb_explore: usb0: first explore done
scsibus1 at sdmmc1: 2 targets, initiator 0
sd1 at scsibus1 targ 1 lun 0: <SD/MMC, USD00, 0010> removable
sd1: 30250MB, 512 bytes/sector, 61952000 sectors
uhub1: port 1 status=0x0500 change=0x0000
usb_explore: usb1: first explore done
uhub2: port 1 status=0x02a0 change=0x0000
uhub2: port 2 status=0x02a0 change=0x0000
usb_explore: usb2: first explore done
uhub3: port 1 status=0x02a0 change=0x0000
uhub3: port 2 status=0x02a0 change=0x0000
usb_explore: usb3: first explore done
uhub4: port 1 status=0x0100 change=0x0000
usb_explore: usb4: first explore done
uhub5: port 1 status=0x0100 change=0x0000
usb_explore: usb5: first explore done
sdmmc0: can't enable card
scsibus2 at sdmmc2: 2 targets, initiator 0
sd2 at scsibus2 targ 1 lun 0: <Sandisk, DA4032, 0000> removable
sd2: 29820MB, 512 bytes/sector, 61071360 sectors
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
ahci0.0: timeout
ahci0.0: port interrupt: 8<SDBS>
ahci0.0: slot 2 is complete
ahci0.0: final poll of port completed command in slot 2
ahci0.0: timeout
ahci0.0: port interrupt: 8<SDBS>
ahci0.0: slot 3 is complete
ahci0.0: final poll of port completed command in slot 3
ahci0.0: timeout
ahci0.0: port interrupt: 8<SDBS>
ahci0.0: slot 4 is complete
ahci0.0: final poll of port completed command in slot 4
ahci0.0: timeout
ahci0.0: port interrupt: 8<SDBS>
ahci0.0: slot 5 is complete
ahci0.0: final poll of port completed command in slot 5
ahci0.0: timeout
ahci0.0: port interrupt: 8<SDBS>
ahci0.0: slot 6 is complete
ahci0.0: final poll of port completed command in slot 6
root on sd2a (2861b74ac6eb9f90.a) swap on sd2b dump on sd2b
rkvop0: using CRTC 0 for RK3399 VOPL
rkvop1: using CRTC 1 for RK3399 VOPB
rkdrm0: 1024x768, 32bpp
wsdisplay0 at rkdrm0 mux 1
wsdisplay0: screen 0-5 added (std, vt100 emulation)
