This series adds support for the Mellanox CX4 network adapter operating in cxl mode to the cxl driver and the PowerNV PHB code. The Mellanox developers will submit a separate patch series that makes use of this in the mlx5 driver.
The CX4 card can operate in either pci mode, or cxl mode. In cxl mode, memory accesses from the card go through the XSL (Translation Service Layer, essentially a stripped down version of the Power Service Layer), allowing it to transparently access unpinned memory with the cxl driver handling faulting in pages as necessary, etc. Most of the support for the XSL is already upstream, though this series does include a bug fix to enable bus mastering for this (patch 3). Patch 2 in this series provides an API which the mlx5 driver can query to check if it is in a cxl capable slot. The card will come up in pci mode, and the mlx5 driver can choose to switch it to cxl mode, wherein it will reappear with an additional physical function representing the XSL that the cxl driver will bind to. Patches 12-14 add support for switching the card's mode, including using the PCI hotplug support to re-enumerate the device tree and re-probind the card. Unlike previous users of the cxl kernel API where we used a virtual PHB and exposed PCI devices under it, the Mellanox CX4 uses a peer model where cxl binds to one of the physical functions of the card and the mlx5_core driver binds to the other networking physical functions. Patches 6 and 7 add support for using the cxl kernel API with the real PHB to enable this peer model. Patches 4 and 5 are prepatory patches exposing some APIs that the PHB will need to call. While in cxl mode, interrupts from the CX4 are a little unusual - they are neither pci interrupts, nor cxl interrutps, but rather a hybrid of the two. The interrupts are passed from the networking hardware to the XSL using a custom format in the MSIX table, and from there are treated as cxl interrupts. These are configured mostly transparently using the standard msix APIs - the PHB handles allocating and configuring the cxl interrupts, associating them with the default context, and the mlx5 driver handles filling out the MSIX table with their custom format (not included in this series). See patch 10. Additionally, the CX4 has a hard limitation of the number of interrupts that can be associated with a given context, so to overcome this patches 8 and 9 expose an API to allow the mlx5 driver to inform us of the limit, and the interrupt allocation code in patch 10 will allocate additional contexts to associate these with. Patch 1 is a prepatory cleanup patch to reorganise cxl code in arch/powerpc into a separate file. Patch 11 is a workaround for a hardware limitation in the CX4 where a context with PE=0 cannot be used. Note that patch 2 depends on "cxl: Ignore CAPI adapters misplaced in switched slot" by Philippe Bergheaud: http://patchwork.ozlabs.org/patch/642920/ Additionally, the following stand-alone patches related to the CX4 are also pending on the mainling list, but are *not* dependencies of this series: - cxl: Fix bug where AFU disable operation had no effect - cxl: Workaround XSL bug that does not clear the RA bit after a reset - cxl: Fix NULL pointer dereference on kernel contexts with no AFU interrupts The entire series is bisectable. _______________________________________________ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
