On 02/01/2016 06:35 AM, David Gibson wrote:
> Hi,
>
> It seems to me we're getting rather bogged down in how to proceed with
> an improved CPU hotplug (and hot unplug) interface, both generically
> and for ppc in particular.
Yes, s390 also needs this.
Can you add Matthew in any cpu hotplug discussion?
>
> So here's a somewhat more concrete suggestion of a way forward, to see
> if we can get some consensus.
>
> The biggest difficulty I think we're grappling with is that device-add
> is actually *not* a great interface to cpu hotplug. Or rather, it's
> not great as the _only_ interface: in order to represent the many
> different constraints on how cpus can be plugged on various platforms,
> it's natural to use a heirarchy of cpu core / socket / package types
> specific to the specific platform or real-world cpu package being
> modeled. However, for the normal case of a regular homogenous (and at
> least slightly para-virtualized) server, that interface is nasty for
> management layers because they have to know the right type to
> instantiate.
>
> To address this, I'm proposing this two layer interface:
>
> Layer 1: Low-level, device-add based
>
> * a new, generic cpu-package QOM type represents a group of 1 or
> more cpu threads which can be hotplugged as a unit
> * cpu-package is abstract and can't be instantiated directly
> * archs and/or individual platforms have specific subtypes of
> cpu-package which can be instantiated
> * for platforms attempting to be faithful representations of real
> hardware these subtypes would match the specific characteristics
> of the real hardware devices. In addition to the cpu threads,
> they may have other on chip devices as sub-objects.
> * for platforms which are paravirtual - or which have existing
> firmware abstractions for cpu cores/sockets/packages/whatever -
> these could be more abstract, but would still be tied to that
> platform's constraints
> * Depending on the platform the cpu-package object could have
> further internal structure (e.g. a package object representing a
> socket contains package objects representing each core, which in
> turn contain cpu objects for each thread)
> * Some crazy platform that has multiple daughterboards each with
> several multi-chip-modules each with several chips, each
> with several cores each with several threads could represent
> that too.
>
> What would be common to all the cpu-package subtypes is:
> * A boolean "present" attribute ("realized" might already be
> suitable, but I'm not certain)
> * A generic means of determining the number of cpu threads in the
> package, and enumerating those
> * A generic means of determining if the package is hotpluggable or
> not
> * They'd get listed in a standard place in the QOM tree
>
> This interface is suitable if you want complete control over
> constructing the system, including weird cases like heterogeneous
> machines (either totally different cpu types, or just different
> numbers of threads in different packages).
>
> The intention is that these objects would never look at the global cpu
> type or sockets/cores/threads numbers. The next level up would
> instead configure the packages to match those for the common case.
>
> Layer 2: Higher-level
>
> * not all machine types need support this model, but I'd expect
> all future versions of machine types designed for production use
> to do so
> * machine types don't construct cpu objects directly
> * instead they create enough cpu-package objects - of a subtype
> suitable for this machine - to provide maxcpus threads
> * the machine type would set the "present" bit on enough of the
> cpu packages to provide the base number of cpu threads
>
> Management layers can then manage hotplug without knowing platform
> specifics by using qmp to toggle the "present" bit on packages.
> Platforms that allow thread-level pluggability can expose a package
> for every thread, those that allow core-level expose a package per
> core, those that have even less granularity expose a package at
> whatever grouping they can do hotplug on.
>
> Examples:
>
> For use with pc (or q35 or whatever) machine type, I'd expect a
> cpu-package subtype called, say "acpi-thread" which represents a
> single thread in the ACPI sense. Toggling those would trigger ACPI
> hotplug events as cpu_add does now.
>
> For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> which represents a single (paravirtual) core. Toggling present on
> this would trigger the PAPR hotplug events. A property would control
> the number of threads in the core (only settable before enabling
> present).
>
> For use with the powernv machine type (once ready for merge) I'd
> expect "POWER8-package" type which represents a POWER8 chip / module
> as close to the real hardware as we can get. It would have a fixed
> number of cores and threads within it as per the real hardware, and
> would also include xscoms and other per-module logic.
>
> From here to there:
>
> A suggested order of implementation to get there without too much risk
> of breaking things.
>
> 1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
> hotplug series already has this)
> 2. Split creation and realization of CPU objects, so machine types
> must explicitly perform both steps (Bharata's series has this
> too)
> 3. Add the abstract cpu-package type, and define the generic
> interfaces it needs (Bharata's series has something that could be
> changed to this fairly easily)
> 4. For each machine type we care to convert:
> 4.1. Add platform suitable cpu-package subtypes
> 4.2. Convert the (latest version) machine type to instantiate packages
> instead of
> cpu threads directly
> 4.3. Add any necessary backwards compat goo
> 5. Teach libvirt how to toggle cpu-packages
>
