These patches are based on spapr-next, and can also be obtained from: https://github.com/mdroth/qemu/commits/spapr-hotplug-pci-v8
v8: * rebased on https://github.com/dgibson/qemu/commits/spapr-next * added sPAPR 2.4 machine-type, updated compat code from 2.3 -> 2.4 * added small helper to lookup a PCI device's DRC more succintly * added additional DPRINTF statements to spapr_drc.c * re-worked how configure-connector handles FDT's (David/Alex): - fdt and fdt_offset for a device are now fetched via a single DRC->get_fdt() method. - state associated with iterating through the FDT via configure-connector call sequences are handled completely in RTAS code (as opposed to within DRC code) via a list of CCState's hung off of sPAPREnvironment (to avoid globals). - a DRC->set_configured() method is added to inform DRC when the FDT has been fetched, which is synonymous with the 'configured' state of the device. - transitions in rtas-set-indicator that unset this 'configured' state in the DRC code now have a hook to also invalidate the corresponding CCState to avoid delivering stale FDT data. v7: * rebased on ppc-next (2015-03-04) * tested/enabled support for multifunction hotplug by allocating a DRC for each device function * spapr_drc: fix "removal deferred" debug statement to only be printed when device is actually awaiting release * spapr_drc: moved all configure-connector handling out, and allow external users (RTAS) to register opaque state and callbacks to deal with init/reset events via drck->get_configure_connector_state() and drck->{begin,complete}_configure_connector() (David) * spapr_pci: use QEMU_PACKED instead of __attribute__((packed)) (David) * spapr_drc: avoid unecessary strdup() when generating DT entries (David) v6: * rebased on ppc-next (2015-02-26) * added S-o-b from Nathan on "spapr_drc: add spapr_drc_populate_dt()" * moved unrelated variable name change up to initial DRC implementation * fixed redundant wording in "spapr_pci: enable basic hotplug operations" * fixed inverted logic on 'assigned-addresses' OF property creation * (packed) ResourceFields to ensure it can be handled as a byte array with the expected sizeof() * dropped union'ing with uint8_t* in favor of a simple cast to handle 'reg'/'assigned-addresses' properties * added comments to clarify need for explicit creation of "name" property * added comments to clarify that 'reg'/'assigned-addresses' are relative to PHB's IO/MEM windows * moved pending rtas event list to sPAPREnvironment * used rtas_error_log->extended length + sizeof(hdr) instead of switching on event type to get event length * renamed pending_epow to new_epow since it suggests only a single pending_epow at any point in time * added an assertion that queued events are non-NULL * re-assert IRQ when RTAS events are still pending * assert that the DRC resource supports hotplug events when callers attempt to queue a hotplug event * added argument validation for rtas-configure-connector * squashed patch to generate PHB device-tree entries for DRCs into the patch that instantiates them, and clarified commit summary v5: * short version: - addressed, to my knowledge, all outstanding review comments from v4 - fixed a number of issues which were made apparent when doing rapid device plug/unplug - allow for a device to be configured/unconfigured/configured reliably without ever being unplugged in between - implemented queueing of RTAS events, and suppression of subsequent unplug events when 1 is already in-flight for a device/DRC. - simplified spapr_drc_populate_dt() and 'reg'/'assigned-addresses' population code to clearer. - fixed OF device node properties in accordance with PCI-OF binding spec - better error-checking/reporting in hotplug path * rebased/retested on ppc-next (2015-02-16) * added support for multiple queued/in-flight EPOW/HP RTAS events * grammar/typo fixes for documentation (David) * fixed bug in drc->set_allocation_state() actually sets allocation state rather than indicator state (David) * ensured all macro expressions were wrapped in parenthesis (David) * dropped drc->get_id() in favor of direct access to drc->id (David) * renamed prop_name to name to avoid naming inconsistencies in drc->configure_connector() (David) * added an assert to ensure QEMU-side users of configure_connector_common() never reach an error condition (David) * rtas-event-scan stub replaced with a functional implementation * ensured unsupported sensor types always result in RTAS_OUT_NOT_SUPPORTED as opposed to RTAS_OUT_PARAM_ERROR (David) * ensured all rtas arg/return counts are validated prior to access (David) * validate power domain is live-insertion/-1 in rtas-{get,set}-power-level (David) * simplified rtas_set_indicator() logic (David) * moved RTAS_SENSOR_TYPE_* macros to spapr.h (David) * modified drc->dr_entity_sense() to report UNUSABLE if the device is logical/non-PCI and the allocation state is UNUSABLE (Bharata) * refactored spapr_drc_populate_dt() to avoid having a separate loop to gather DRC fields into a temporary data structure prior to generating OF array properties (David) * added an Object *owner fields to spapr_drc_populate_dt() to allow limiting of OF DRC/slot descriptions to specific PHBs (David) * ensure true/false are used in place of 1/0 (David) * make naming of hotplug/unplug hooks clearer (David) * re-worked 'reg'/'assigned-address' OF property population to avoid potential buffer-overrun and make it easier to understand the purpose of individual fields. also added documentation to further clarify logic (David) * fixed boolean OF device properties to be present-but-empty/not-present rather than storing corresponding device register bits (David) * fixed subsystem-id/subsystem-vendor-id OF properties to only be present when non-zero * use actual bus num in place of phb index for 'reg'/'assigned-address' fields and DRC indexes v4: * added documentation for sPAPR-based hotplug (Alexey) * reworked DR Connectors to be QOM devices, where sensor/indicator states are accessed via RTAS via object methods and also exposed via composition tree for introspection via qom-get/qom-fuse. attached devices are managed via state transitions handled by the DRC device (Alex) * DRC-related constants now defined in seperate header file, implemented as enum types where applicable * removed stub implementations of sensors that were not relevant to dynamic-reconfiguration. we now return "not implemented" if a guest attempts to access them via rtas-get-sensor or rtas-set-indicator-state * added DRC reset hooks to complete unplug for devices awaiting additional action from the guest before removal * incorporated endian fixes from Bharata and tested on ppc64le (Alex/Bharata) * used rtas_{ld,st} helpers in place of cpu_physical_memory_map for configure-connector implementation (Alex) * used b_* helper macros for properties related to OF PCI Binding (Alexey) * added dynamic-reconfiguration option to spapr-pci-host-bridge to enable/disable PCI hotplug for child bus * added pseries-2.3 machine and compat code to disable PCI hotplug by default for older machine types (Alex) * removed OF properties and DRC instances related to hotplugging of PHBs. this is not a prereq for PCI hotplug and will be handled as a separate series * moved generation of boot-time devices properties to common helper that can be re-used for memory, cpu, and phb. (Bharata) * re-organized patches so that pci, memory, cpu, phb should base cleanly on common set of patches implementing core DRC functionality (Bharata) * moved PCI 0-address fix to separate series (Alex) v3: * dropped emulation of firmware-managed BAR allocation. this will be introduced via a follow-up series via a -machine flag and tied to a separate hotplug event to avoid a race condition with guest vs. "firmware"-managed BAR allocation, in conjunction with required fixes to rpaphp hotplug kernel module to utilize this mode. * moved drc_table into sPAPREnvironment (Alexey) * moved INDICATOR_* constants and friends into spapr_pci.c (Alexey) * use prefixes for global types (DrcEntry/ConfigureConnectorState) (Alexey) * updated for new hotplug interface (Alexey) * fixed get-power-level to report current power-level rather than desired (Alexey) * rebased to latest ppc-next v2: * re-ordered patches to fix build bisectability (Alexey) * replaced g_warning with DPRINTF in RTAS calls for guest errors (Alexey) * replaced g_warning with fprintf for qemu errors (Alexey) * updated RTAS calls to use pre-existing error/success macros (Alexey) * replaced DR_*/SENSOR_* macros with INDICATOR_* for set-indicator/ get-sensor-state (Alexey) OVERVIEW These patches add support for PCI hotplug for SPAPR guests. We advertise each PHB as DR-capable (as defined by PAPR 13.5/13.6) with 32 hotpluggable PCI slots per PHB, which models a standard PCI expansion device for Power machines where the DRC name/loc-code/index for each slot are generated based on bus/slot number. This is compatible with existing guest kernel's via the rpaphp hotplug module, and existing userspace tools such as drmgr/librtas/rtas_errd for managing devices. NOTES / ADDITIONAL DEPENDENCIES This series relies on v1.2.19 or later of powerppc-utils (drmgr, rtas_errd, ppc64-diag, and librtas components, specificially), which will automate guest-side hotplug setup in response to an EPOW event emitted by QEMU. For guests with older versions of powerpc-utils, a manual workaround must be used (documented below). Note that this relies on a patch to core PCI code which allows for the use of a 0-address IO BAR for PCI devices. Without this patch, the first hotplugged device will likely fail. This patch will be handled separately, but is included in the in the development tree below for testing: https://github.com/mdroth/qemu/commits/spapr-hotplug-pci PATCH LAYOUT Patches 1 Documentation for sPAPR Dynamic-Reconfiguration/hotplug 2 Initial implementation for sPAPRDRConnector device 3-7 Guest RTAS calls to interact with DRC devices 8-9 Introduce RTAS events for signalling hotplug operations to guest, using existing infrastructure of EPOW/check-exception events 10 DRC helper code to populate DT descriptions of present DRC devices 11 introduce spapr-2.4 machine 12 spapr-host-bridge option to selectively enable PCI hotplug/DR on a PHB-by-PHB basis 13-16 PCI-specific hotplug hooks and DRC creation to enable PCI hotplug and hotplug events USAGE For guests with powerpc-utils 1.2.19+: hotplug: qemu: device_add e1000,id=slot0 unplug: qemu: device_del slot0 For guests with powerpc-utils prior to 1.2.19: hotplug: qemu: device_add e1000,id=slot0 guest: drmgr -c pci -s "C0" -n -a echo 1 >/sys/bus/pci/rescan unplug: guest: drmgr -c pci -s "C0" -n -r echo 1 >/sys/bus/pci/devices/0000:00:00.0/remove qemu: device_del slot0 docs/specs/ppc-spapr-hotplug.txt | 287 +++++++++++++ hw/pci/pci.c | 2 +- hw/ppc/Makefile.objs | 2 +- hw/ppc/spapr.c | 41 +- hw/ppc/spapr_drc.c | 743 +++++++++++++++++++++++++++++++++ hw/ppc/spapr_events.c | 337 ++++++++++++--- hw/ppc/spapr_pci.c | 412 +++++++++++++++++- hw/ppc/spapr_rtas.c | 360 ++++++++++++++++ include/hw/pci-host/spapr.h | 1 + include/hw/pci/pci.h | 1 + include/hw/ppc/spapr.h | 52 ++- include/hw/ppc/spapr_drc.h | 201 +++++++++ 12 files changed, 2372 insertions(+), 67 deletions(-)
