I've been evaluating some new options for our POWER9-based hardware in the NVMe space, and would like some clarification on the current status of PCIe hotplug for the PowerNV platforms.
>From what I understand, the pnv_php driver provides the basic hotplug >functionality on PowerNV. What I'm not clear on is to what extent this is >intended to flow downstream to attached PCIe switches. I have a test setup here that consists of a PMC 8533 switch and several downstream NVMe drives, with the switch attached directly to the PHB4 root port. After loading the pnv_php module, I can disconnect the downstream NVMe devices by either using echo 0 on /sys/bus/pcu/slots/Snnnnnnn/power, or by doing a physical surprise unplug, however nothing I try can induce a newly plugged device to train and be detected on the bus. Even trying a echo 0 and subsequent echo 1 to /sys/bus/pcu/slots/Snnnnnnn/power only results in the device going offline, there seems to be no way to bring the device back online short of a reboot. Hotplug of other devices connected directly to the PHB4 appears to work properly (I can online and offline via the power node); the issue seems to be restricted to downstream devices connected to the (theoretically hotplug capable) PMC 8533 switch. Is this the intended behavior for downstream (non-IBM) PCIe ports? Raptor can provide resources to assist in a fix if needed, but I would like to understand if this is a bug or an unimplemented feature first, and if the latter what the main issues are likely to be in implementation. Thank you!
