Hi,
This series is a first part of the implementation of CPU topology
for S390.
====================
A short introduction
====================
CPU Topology is described in the S390 POP with essentially the description
of two instructions:
PTF Perform Topology function used to poll for topology change
and used to set the polarization but this part is not part of this item.
STSI Store System Information and the SYSIB 15.1.x providing the Topology
configuration.
S390 Topology is a 6 levels hierarchical topology with up to 5 level
of containers. The last topology level, specifying the CPU cores.
To get the information on the topology, S390 provides the STSI
instruction, which stores a structures providing the list of the
containers used in the Machine topology: the SYSIB.
A selector within the STSI instruction allow to chose how many topology
levels will be provide in the SYSIB.
Using the Topology List Entries (TLE) provided inside the SYSIB we
the Linux kernel is able to compute the information about the cache
distance between two cores and can use this information to take
scheduling decisions.
We first provide a simple topology for the case QEMU does not
use NUMA by extending the -smp argument and then we provide more
fine grain topology in the case QEMU uses the -numa arguments.
Note:
-----
Z15 reports 3 levels of containers, drawers, book, sockets as
Container-TLEs above the core description inside CPU-TLEs.
The Topology can be seen at several places inside zLinux:
- sysfs: /sys/devices/system/cpu/cpuX/topology
- procfs: /proc/sysinfo and /proc/cpuinfo
- lscpu -e : gives toplogy information
The different Topology levels have names:
- Node - Drawer - Book - sockets or physical package - core
Threads:
Multithreading, is not part of the topology as described by the
SYSIB 15.1.x
The interest of the guest to know the CPU topology is obviously to be
able to optimise the load balancing and the migration of threads.
KVM will have the same interest concerning vCPUs scheduling and cache
optimisation.
====================
The design
====================
1) To be ready for hotplug, I chose an Object oriented design
of the topology containers:
- A node is a bridge on the SYSBUS and defines a "node bus"
- A drawer is hotplug on the "node bus"
- A book on the "drawer bus"
- A socket on the "book bus"
- And the CPU Topology List Entry (CPU-TLE)sits on the socket bus.
These objects will be enhanced with the cache information when
NUMA is implemented.
This also allows for easy retrieval when building the different SYSIB
for Store Topology System Information (STSI)
2) Perform Topology Function (PTF) instruction is made available to the
guest with a new KVM capability and intercepted in QEMU, allowing the
guest to pool for topology changes.
======================
Current implementation
======================
* qemu command line is extended with the new topology levels.
Here a comparison with X86
for X86:
-smp
[cpus=]n[,cores=cores][,threads=threads][,dies=dies][,sockets=sockets][,maxcpus=maxcpus]
old S390:
-smp [cpus=]n[,sockets=sockets][,cores=cores][,maxcpus=maxcpus]
new S390:
-smp
[cpus=]n[,drawers=drawers][,books=books][,sockets=sockets][,cores=cores][,maxcpus=maxcpus]
Example:
--------
Here we want to use all cores on book 3, on socket-id 2 and the cores 0 and 1:
/usr/local/bin/qemu-system-s390x \
-machine s390-ccw-virtio,accel=kvm \
-enable-kvm \
-m 10G \
\
-smp 15,drawers=5,books=2,sockets=2,cores=12,maxcpus=240 \
\
-object memory-backend-ram,id=mem0,size=2G \
-object memory-backend-ram,id=mem1,size=2G \
-object memory-backend-ram,id=mem2,size=2G \
-object memory-backend-ram,id=mem3,size=2G \
-object memory-backend-ram,id=mem4,size=2G \
\
-numa node,nodeid=0,memdev=mem0\
-numa node,nodeid=1,memdev=mem1\
-numa node,nodeid=2,memdev=mem2\
-numa node,nodeid=3,memdev=mem3\
-numa node,nodeid=4,memdev=mem4\
\
-numa cpu,node-id=1,core-id=0 \
-numa cpu,node-id=4,core-id=1 \
-numa cpu,node-id=2,socket-id=2 \
-numa cpu,node-id=3,book-id=3 \
\
-netdev tap,id=hn0,queues=1 \
\
-device virtio-net-ccw,netdev=hn0,mq=on \
-kernel /boot/vmlinuz-${YOUR_KERNEL} \
-initrd ${YOUR_ROOTFS} \
-append "loglevel=8 selinux=0 root=/dev/ram earlyprintk=1" \
-nographic
=====================
Features and TBD list
=====================
- using a default memory device
- There is a warning about all CPU should be described in NUMA config and that
this would be obsolete in the future, I will need help to know how to
handle this.
- There is no direct match between IDs shown by:
- lscpu (unrelated numbered list),
- used by -numa cpu (numbered list related to topology)
- SYSIB 15.1.x (topology ID)
in the example above socket-id=6 is shown in SYSIB 15.1.x as
socket ID 0 of Book ID 3 since a Book has 2 sockets and appear
on lscpu as socket 2.
- The CPU number, left column of lscpu, is used to reference a CPU
by Linux tools
While the CPU address is used by QEMU for hotplug.
- To plug a new CPU inside the topology one can simply use the CPU
address like in:
(qemu) device_add host-s390x-cpu,core-id=8
CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION
ADDRESS
0 0 0 0 0 0 0:0:0:0 yes yes horizontal 0
1 0 0 1 1 1 1:1:1:1 yes yes horizontal
24
2 0 0 1 1 2 2:2:2:2 yes yes horizontal
25
....
36 0 1 3 4 36 36:36:36:36 yes yes horizontal
94
37 0 1 3 4 37 37:37:37:37 yes yes horizontal
95
38 - - - - - ::: no yes horizontal
8
# chcpu -e 38
- Documentation will come with the next iteration
Regards,
Pierre
Pierre Morel (9):
s390x: smp: s390x dedicated smp parsing
s390x: toplogy: adding drawers and books to smp parsing
s390x: cpu topology: CPU topology objects and structures
s390x: Topology list entries and SYSIB 15.x.x
s390x: topology: implementating Store Topology System Information
s390x: kvm: topology: interception of PTF instruction
s390x: SCLP: reporting the maximum nested topology entries
s390x: numa: define drawers and books for NUMA
s390x: numa: implement NUMA for S390x
hw/core/machine.c | 18 +
hw/s390x/cpu-topology.c | 595 +++++++++++++++++++++++++++++
hw/s390x/meson.build | 1 +
hw/s390x/s390-virtio-ccw.c | 107 +++++-
hw/s390x/sclp.c | 1 +
include/hw/s390x/cpu-topology.h | 99 +++++
include/hw/s390x/s390-virtio-ccw.h | 3 +
include/hw/s390x/sclp.h | 4 +-
qapi/machine.json | 2 +
softmmu/vl.c | 6 +
target/s390x/cpu.c | 4 +
target/s390x/cpu.h | 45 +++
target/s390x/kvm/kvm.c | 273 +++++++++++++
13 files changed, 1150 insertions(+), 8 deletions(-)
create mode 100644 hw/s390x/cpu-topology.c
create mode 100644 include/hw/s390x/cpu-topology.h
--
2.25.1
