Re: boot interrupt quirk (also in 4.19.y) breaks serial ports (was: [PATCH v2 0/2] pci: Add boot interrupt quirk mechanism for Xeon chipsets)
Hi tglx, On 11/25/20 12:54 PM, Thomas Gleixner wrote: > Stefan, > > On Wed, Sep 16 2020 at 12:12, Stefan Bühler wrote: > > sorry for the delay. This fell through the cracks. > >> this quirk breaks our serial ports PCIe card (i.e. we don't see any >> output from the connected devices; no idea whether anything we send >> reaches them): >> >> 05:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI >> Bridge (rev aa) >> 06:00.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 >> UART) function 0 (Uart) >> 06:00.1 Bridge: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) >> function 0 (Disabled) >> 06:01.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 >> UART) function 0 (Uart) >> 06:01.1 Bridge: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) >> function 0 (Disabled) > > Can you please provide the output of: > > for ID in 05:00.0 06:00.0 06:00.1 06:01.0 06:01.1; do lspci -s $ID -vvv; done > See attachment. Also I boot the affected systems now with "pci=noioapicquirk", which "solves" the issue too (instead of patching the kernel). cheers, Stefan 05:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) Physical Slot: 1 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: 06:00.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) (prog-if 06 [16950]) Subsystem: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- Kernel driver in use: serial 06:00.1 Bridge: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Disabled) Subsystem: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Disabled) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- 06:01.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) (prog-if 06 [16950]) Subsystem: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- Kernel driver in use: serial 06:01.1 Bridge: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Disabled) Subsystem: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Disabled) Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR-
Re: boot interrupt quirk (also in 4.19.y) breaks serial ports (was: [PATCH v2 0/2] pci: Add boot interrupt quirk mechanism for Xeon chipsets)
Hi tglx, On 11/27/20 12:45 AM, Thomas Gleixner wrote: > Stefan, > > On Wed, Nov 25 2020 at 14:41, Stefan Bühler wrote: >> On 11/25/20 12:54 PM, Thomas Gleixner wrote: >>> On Wed, Sep 16 2020 at 12:12, Stefan Bühler wrote: >>> Can you please provide the output of: >>> >>> for ID in 05:00.0 06:00.0 06:00.1 06:01.0 06:01.1; do lspci -s $ID -vvv; >>> done >> >> 05:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI >> Bridge (rev aa) (prog-if 00 [Normal decode]) >> ... >> Capabilities: > > Can you please run this as root so the Capabilities are accessible? My bad, sorry. I did intend to run it as root, but should have checked the output. Again see attached file. While we're at it: the EEPROM for the PEX8112 is: 5a 03 3c 00 10 00 00 00 00 00 00 00 b5 10 12 81 |Z.<.| 0010 64 00 20 00 00 00 00 01 04 00 01 00 0c 10 00 fe |d. .| 0020 fe 03 20 10 f0 10 00 00 00 10 33 00 00 00 70 00 |.. ...3...p.| 0030 00 00 11 00 48 00 00 00 00 00 34 00 50 00 00 00 |H.4.P...| 0040 04 00 55 66 77 88 |..Ufw.| 0046 (This is what the firmware tool provided to me writes, although I think the cards usually came pre-flashed with it. They gave me the tool because on some cards the second function on OX16PCI954 was sometimes uninitialized, came up with device id 0x9511 "8-bit bus" (PCI_DEVICE_ID_OXSEMI_16PCI95N) and the kernel tries to treat it as UART too.) I think some time ago I found a PDF to decode this here: https://www.broadcom.com/products/pcie-switches-bridges/pcie-bridges/pex8112#documentation But the broadcom site is completely broken right now (at least for me; there own search for "PEX 8112" links it, but then it says "not found"). Anyway, back then I decoded this to: - `0x5A 0x03`: Magic Header, contains register and shared memory settings - `0x003C` = 60 bytes for configs (10 registers): - `@0x0010`: `0x` -- BAR0: Locate anywhere in 32-bit - `@0x`: `0x811210B5` -- Vendor `10B5`, Device `8112` (default) - `@0x0064`: `0x0020` -- Device Capability: Enable "Support 8-bit Tag" field - `@0x0100`: `0x00010004` -- Power Budget Enhanced Capability Header (default) - `@0x100C`: `0x03FEFE00` -- PCI Control: - PCI-To-PCI Express Retry Count set to 0xFE (default: `0x80`) - PCI Express-to-PCI Retry Count set to 0xFE (default: `0x00`) - `@0x1020`: `0x10F0` -- GPIO Control - GPIO[1-3] Output enable (GPIO[0] is Output enabled by default) - GPIO Diagnostic Select: `10b` (default: `01b`) - `@0x1000`: `0x0033` -- Device Initialization (default) - `@0x0070`: `0x0011` -- Link control: default - `@0x0048`: `0x` -- Device-Specific Control (default 0) - `@0x0034`: `0x0050` -- PCI Capability pointer `0x50` (default: `0x40`) - Skips (disables) Power Management Capability - Remaining: MSI and PCI Express - `0x0004` bytes for shared memory: - `0x55`, `0x66`, `0x77`, `0x88` TLDR: the most notable part probably being "disabling Power Management Capability" by the EEPROM. cheers, Stefan 05:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) (prog-if 00 [Normal decode]) Physical Slot: 1 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: Data: Capabilities: [60] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0 ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 26.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop- BrConfRtry- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [100 v1] Power Budgeting 06:00.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) (prog-if 06 [16950])
boot interrupt quirk (also in 4.19.y) breaks serial ports (was: [PATCH v2 0/2] pci: Add boot interrupt quirk mechanism for Xeon chipsets)
Hi, this quirk breaks our serial ports PCIe card (i.e. we don't see any output from the connected devices; no idea whether anything we send reaches them): 05:00.0 PCI bridge: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge (rev aa) 06:00.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) 06:00.1 Bridge: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Disabled) 06:01.0 Serial controller: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Uart) 06:01.1 Bridge: Oxford Semiconductor Ltd OX16PCI954 (Quad 16950 UART) function 0 (Disabled) The hardware is "MP8-PCIe-RS232-R2" (we also have the 16-port variant) from https://rs-232.de/sercom_e.htm#MP8PCIERS232 The CPU is an "Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz". Disabling the quirk for 0x6f28 (Xeon D-1500) on 4.19.145 gets the serial ports working again (patch attached for reference). The box is running debian stable; we build ixgbe 5.6.1 through dkms (local package) due to issues with the SFP+ ports (usually our uplink...). Known broken versions: 4.19.115-00065, 4.19.118, 4.19.145, 5.8.9 Known good versions: 4.19.115-00064 Basically if we see "kernel: pci :00:05.0: disabled boot interrupts on device [8086:6f28]" in the kernel log it's broken. Attached lspci, lscpu, lsmod and kernel log outputs while running 5.8.9. Can we disable the quirk somehow via the kernel commandline? Anything else I can do to help getting this fixed properly? (We have the same card in other boxes with CPUs not affected by this quirk; they are working just fine.) cheers, Stefan -- Stefan BühlerMail/xmpp: stefan.bueh...@tik.uni-stuttgart.de Netze und Kommunikationssysteme der Universität Stuttgart (NKS) https://www.tik.uni-stuttgart.de/Telefon: +49 711 685 60854 00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 03) 00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03) 00:01.1 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 03) 00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03) 00:02.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 03) 00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03) 00:03.2 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 03) 00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 03) 00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 03) 00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 03) 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05) 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04) 00:16.1 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #2 (rev 04) 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5) 00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5) 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5) 00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5) 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05) 00:1f.0 ISA bridge: Intel Corporation C224 Series Chipset Family Server Standard SKU LPC Controller (rev 05) 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05) 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05) 01:00.0 Non-Volatile memory controller: Intel Corporation SSD 600P Series (rev 03) 03:00.0 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 0 03:00.1 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 1 03:00.2 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 2 03:00.3 System peripheral: Intel Corporation Xeon Processor D Family QuickData Technology Register DMA Channel 3 04:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552 10 GbE SFP+ 04:00.1 Ethernet controller: Intel Corporation Ethernet Connection X
