This is v3 of the CPU and Memory hotplug patches for PowerPC sPAPR.
- CPU hotplug implementation here is based on device_add command and has
nothing to do with the existing cpu-add QEMU monitor command.
(qemu) device_add powerpc64-cpu-socket,id=sock1
(qemu) device_del sock1
- This version adds a full socket in response to device_add. Though for
PowerPC, the term socket really doesn't make sense, it is used as a
container of CPU cores here. Unless sockets= option is explicitly specified,
QEMU assigns one core per socket by default for PowerPC and the hotplug
implementation too sticks to this behaviour.
- Currently hotplugging a socket populates the cores and threads based on
the topology specified at boot time. i,e., there isn't a way to have add
partially populated sockets resulting in non-homogeneous configurations.
I have kept it this way since I don't want to invent and implement
any new semantics that hasn't been agreed upon.
- Andreas has been mentioning about having link<> properties for the machine
defined at boot time and populated at hotplug time by CPU objects that get
created. I am not following that here as its need wasn't felt.
- The CPU hotplug semantics discussion is still underway but I am posting
this version with v2 review comments addressed and more fixes incorporated.
Changes in v3
-------------
CPU
---
- Minor reworks on CPU device tree code reorganzation. Refer to patch
description in 05/24 to see how the DT generation code flow looks like
from different call sites.
- Don't create ibm,my-drc-index property when CPU DR isn't enabled. (12/24)
- Enable reuse of device ID string after unplug by releasing core and socket
object when a vCPU is destroyed. (17/24) (Removed David's Reviewed-by
from 17/24 since the patch got changed)
- Releasing cores and sockets during vcpu destroy needed a new QOM API (16/24).
- Deduce socket, core and thread numbers correctly even when none or only
some of them are explicitly specified. (11/24)
- Remove vCPU thread only after sending the unplug notification to guest and
getting ACK from it. (20/24)
- Add CPU device coldplug support which allows CPU to be specified on QEMU
cmdline like -device powerpc64-cpu-socket,id=sock1. This is in addition
to the CPUs that are specified using -smp X. Coldplug support is needed
for migration. (12/24)
- Setup CPU DR connectors early before boot CPUs are setup. This is needed
to support coldplug. (02/24 - Removed David's Reviewed-by since the patch
got changed)
- Ensure hot and cold plug of CPUs are handled correctly for machines where
CPU DR isn't enabled. (12/24)
- Ensure NUMA node information is set for cold pluggged CPUs too. (12/24)
Memory
------
- Use ldl_phys instead of rtas_ld as suggested by David. (23/24)
- Support cold plugging of memory device via QEMU cmdline like
-object memory-backend-ram,id=ram1,size=1G -device pc-dimm,id=dimm,memdev=ram1
Ensure SPAPR_LMB_FLAGS_ASSIGNED flags are set for cold-plugged memory
too. Cold plugging support enables migration. (23/24)
- New API to lookup NUMA node by address. (22/24)
- Removed unused enforced-aligned-dimm property.
- Enforce memory hotplug to node 0 only since pseries kernel still doesn't
allow updation of associativity index for hotplugged LMBs. (24/24)
- Do vm_unregister_ram() and memory_region_del_subregion() when hot adding
of memory fails. (24/24)
- Fail hotplug and unplug for machine versions that don't support memory
DR (24/24)
- Resorted to fdt_setprop instead of fdt_setprop64 since the latter isn't
present yet in the DTC submodule of QEMU. (23/24)
- Use uint64_t for lmb_size so that memory > 4G is handled correctly by
the guest (23/24)
Known issues and limitations
----------------------------
- Hot removal of CPUs in random order breaks migration. Fixing this needs
invention of new semantics.
- Still not able to fail the hotplug and do proper cleanup for hot plug
requests on machines which don't support CPU DR. Resorting to silent
failure without raising an error to the user.
- Memory can be hotplugged to Node 0 only currently. IIUC, to support hotplug
to other NUMA nodes, guest kernel support for sending configure-connector
call during hotplug is needed.
- Guest NUMA nodes come up with flat distance after supporting
ibm,dynamic-reconfiguration-memory node. Still debugging this.
- DRC states are still not migrated, this is needed to support migration
with hotplug correctly. Without it there are limitations:
- VM hotplugged with CPUs can be migrated and hotplugged CPUs
can be removed at the target (supported by a hack)
- VM hotplugged with memory can be migrated however hot memory
addition to a migrated VM isn't possible yet.
These oddities will go away after DRC state migration is supported.
- David's long standing review comment about sharing code b/n x86 and
PowerPC memory hotplug handlers is yet to be addressed.
Dependencies
------------
- For CPU and Memory hotplug to work, latest powerpc-utils and ppc64-diag
packages are needed in the guest. version 1.2.25 of powerpc-utils
and latest git master (unreleased) of ppc64-diag are required.
Previous versions
-----------------
v2: http://lists.nongnu.org/archive/html/qemu-devel/2015-03/msg04737.html
v1: http://lists.gnu.org/archive/html/qemu-devel/2015-01/msg00611.html
v0: http://lists.nongnu.org/archive/html/qemu-devel/2014-09/msg00752.html
Git tree for this implementation
--------------------------------
- These patches apply against spapr-hotplug-pci-v7 branch of Michael Roth's
PCI hotplug tree (git://github.com/mdroth/qemu)
- The current patchset can be fetched from
spapr-hotplug branch at https://github.com/bharata/qemu/
Andreas Färber (1):
cpu: Prepare Socket container type
Bharata B Rao (21):
spapr: Add DRC dt entries for CPUs
spapr: Consider max_cpus during xics initialization
spapr: Support ibm,lrdr-capacity device tree property
spapr: Reorganize CPU dt generation code
spapr: Consolidate cpu init code into a routine
ppc: Prepare CPU socket/core abstraction
spapr: Add CPU hotplug handler
ppc: Update cpu_model in MachineState
ppc: Create sockets and cores for CPUs
spapr: CPU hotplug support
cpus: Add Error argument to cpu_exec_init()
cpus: Convert cpu_index into a bitmap
ppc: Move cpu_exec_init() call to realize function
qom: Introduce object_has_no_children() API
xics_kvm: Don't enable KVM_CAP_IRQ_XICS if already enabled
xics_kvm: Add cpu_destroy method to XICS
spapr: CPU hot unplug support
spapr: Initialize hotplug memory address space
numa: API to looking NUMA node by address
spapr: Support ibm,dynamic-reconfiguration-memory
spapr: Memory hotplug support
Gu Zheng (1):
cpus, qom: Reclaim vCPU objects
Michael Roth (1):
spapr: enable PHB/CPU/LMB hotplug for pseries-2.3
cpus.c | 67 +++
default-configs/ppc64-softmmu.mak | 1 +
docs/specs/ppc-spapr-hotplug.txt | 66 +++
exec.c | 39 +-
hw/cpu/Makefile.objs | 2 +-
hw/cpu/socket.c | 21 +
hw/intc/xics.c | 12 +
hw/intc/xics_kvm.c | 19 +
hw/ppc/Makefile.objs | 1 +
hw/ppc/cpu-core.c | 65 +++
hw/ppc/cpu-socket.c | 68 +++
hw/ppc/mac_newworld.c | 10 +-
hw/ppc/mac_oldworld.c | 7 +-
hw/ppc/ppc440_bamboo.c | 7 +-
hw/ppc/prep.c | 7 +-
hw/ppc/spapr.c | 1054 ++++++++++++++++++++++++++++++-------
hw/ppc/spapr_events.c | 11 +-
hw/ppc/spapr_hcall.c | 51 +-
hw/ppc/spapr_rtas.c | 29 +-
hw/ppc/virtex_ml507.c | 7 +-
include/exec/exec-all.h | 2 +-
include/hw/cpu/socket.h | 14 +
include/hw/ppc/cpu-core.h | 32 ++
include/hw/ppc/cpu-socket.h | 32 ++
include/hw/ppc/spapr.h | 37 +-
include/hw/ppc/xics.h | 3 +
include/qom/cpu.h | 19 +
include/qom/object.h | 11 +
include/sysemu/kvm.h | 1 +
include/sysemu/numa.h | 3 +
kvm-all.c | 57 +-
kvm-stub.c | 5 +
numa.c | 61 +++
qom/object.c | 12 +
target-alpha/cpu.c | 8 +-
target-arm/cpu.c | 3 +-
target-cris/cpu.c | 8 +-
target-i386/cpu.c | 8 +-
target-lm32/cpu.c | 8 +-
target-m68k/cpu.c | 8 +-
target-microblaze/cpu.c | 8 +-
target-mips/cpu.c | 8 +-
target-moxie/cpu.c | 8 +-
target-openrisc/cpu.c | 8 +-
target-ppc/cpu.h | 1 +
target-ppc/translate_init.c | 65 ++-
target-s390x/cpu.c | 3 +-
target-sh4/cpu.c | 8 +-
target-sparc/cpu.c | 3 +-
target-tricore/cpu.c | 7 +-
target-unicore32/cpu.c | 8 +-
target-xtensa/cpu.c | 8 +-
52 files changed, 1747 insertions(+), 264 deletions(-)
create mode 100644 hw/cpu/socket.c
create mode 100644 hw/ppc/cpu-core.c
create mode 100644 hw/ppc/cpu-socket.c
create mode 100644 include/hw/cpu/socket.h
create mode 100644 include/hw/ppc/cpu-core.h
create mode 100644 include/hw/ppc/cpu-socket.h
--
2.1.0
