On Thu, Dec 28, 2023 at 3:16 PM Timothy Pearson <tpear...@raptorengineering.com> wrote: > > I've been evaluating some new options for our POWER9-based hardware in the > NVMe space, and would like some clarification on the current status of PCIe > hotplug for the PowerNV platforms. > > From what I understand, the pnv_php driver provides the basic hotplug > functionality on PowerNV. What I'm not clear on is to what extent this is > intended to flow downstream to attached PCIe switches.
I did a bunch of work on NVMe hotplug back in the day and it worked fine then. Most of that work was done with Gen3 PLX switches though which are considerably dumber than the newer ones though. > I have a test setup here that consists of a PMC 8533 switch and several > downstream NVMe drives, with the switch attached directly to the PHB4 root > port. After loading the pnv_php module, I can disconnect the downstream NVMe > devices by either using echo 0 on /sys/bus/pcu/slots/Snnnnnnn/power, or by > doing a physical surprise unplug, however nothing I try can induce a newly > plugged device to train and be detected on the bus. Even trying a echo 0 and > subsequent echo 1 to /sys/bus/pcu/slots/Snnnnnnn/power only results in the > device going offline, there seems to be no way to bring the device back > online short of a reboot. > > Hotplug of other devices connected directly to the PHB4 appears to work > properly (I can online and offline via the power node); the issue seems to be > restricted to downstream devices connected to the (theoretically hotplug > capable) PMC 8533 switch. I'd suspect either the PCIe interrupt is not being delivered for some reason (EEH might be isolating the PCIe switch port?) or the removal is triggering downstream port containment on the switch. Check the port capability status with lspci. IIRC pnv_php doesn't know anything about DPC so you might need to have skiboot disable that by default to keep the kernel happy. > Is this the intended behavior for downstream (non-IBM) PCIe ports? Raptor > can provide resources to assist in a fix if needed, but I would like to > understand if this is a bug or an unimplemented feature first, and if the > latter what the main issues are likely to be in implementation. It *should* work and the WARN()/BUG() spew you're seeing are bugs that just need fixing. That said, hotplug on PNV is a headache for a bunch of reasons most of which are due to EEH. Something I was working towards before I left IBM was refactoring how EEH worked internally so we could eliminate the need for pnv_php and go back to using the standard pcieport driver. Unfortunately, that's a big job and I can't even remember how much of that work actually made it upstream.
