Hi,
I've stumbled in a LMB hot unplug problem when running a guest with
4.13+ kernel using QEMU 2.10. When trying to hot unplug a recently
hotplugged LMB this is what I got, using an upstream kernel:
---------------
QEMU cmd line: sudo ./qemu-system-ppc64 -name migrate_qemu -boot
strict=on -device nec-usb-xhci,id=usb,bus=pci.0,addr=0xf -device
spapr-vscsi,id=scsi0,reg=0x2000 -smp
32,maxcpus=32,sockets=32,cores=1,threads=1 --machine
pseries,accel=kvm,kvm-type=HV,usb=off,dump-guest-core=off -m
4G,slots=32,maxmem=32G -drive
file=/home/danielhb/vm_imgs/f26.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x2,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-nographic
Last login: Wed Oct 4 12:28:25 on hvc0
[danielhb@localhost ~]$ grep Mem /proc/meminfo
MemTotal: 4161728 kB
MemFree: 3204352 kB
MemAvailable: 3558336 kB
[danielhb@localhost ~]$ QEMU 2.10.50 monitor - type 'help' for more
information
(qemu)
(qemu) object_add memory-backend-ram,id=ram0,size=1G
(qemu) device_add pc-dimm,id=dimm0,memdev=ram0
(qemu)
[danielhb@localhost ~]$ grep Mem /proc/meminfo
MemTotal: 5210304 kB
MemFree: 4254656 kB
MemAvailable: 4598144 kB
[danielhb@localhost ~]$ (qemu)
(qemu) device_del dimm0
(qemu) [ 136.058727] pseries-hotplug-mem: Memory indexed-count-remove
failed, adding any removed LMBs
(qemu)
[danielhb@localhost ~]$ grep Mem /proc/meminfo
MemTotal: 5210304 kB
MemFree: 4253696 kB
MemAvailable: 4597184 kB
[danielhb@localhost ~]$
-------------
The LMBs are about to be unplugged, them something happens and they
ended up being hotplugged back.
This isn't reproducible with 4.11 guests. I can reliably reproduce it in
4.13+. Haven't tried 4.12.
Changing QEMU snapshots or even the hypervisor kernel/OS didn't affect
the result.
In an attempt to better understand the issue I did the following changes
in upstream kernel:
diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c
b/arch/powerpc/platforms/pseries/hotplug-memory.c
index 1d48ab424bd9..37550833cdb0 100644
--- a/arch/powerpc/platforms/pseries/hotplug-memory.c
+++ b/arch/powerpc/platforms/pseries/hotplug-memory.c
@@ -433,8 +433,10 @@ static bool lmb_is_removable(struct of_drconf_cell
*lmb)
unsigned long pfn, block_sz;
u64 phys_addr;
- if (!(lmb->flags & DRCONF_MEM_ASSIGNED))
+ if (!(lmb->flags & DRCONF_MEM_ASSIGNED)) {
+ pr_err("lmb is not assigned \n");
return false;
+ }
block_sz = memory_block_size_bytes();
scns_per_block = block_sz / MIN_MEMORY_BLOCK_SIZE;
@@ -442,8 +444,10 @@ static bool lmb_is_removable(struct of_drconf_cell
*lmb)
#ifdef CONFIG_FA_DUMP
/* Don't hot-remove memory that falls in fadump boot memory area */
- if (is_fadump_boot_memory_area(phys_addr, block_sz))
+ if (is_fadump_boot_memory_area(phys_addr, block_sz)) {
+ pr_err("lmb belongs to fadump boot memory area\n");
return false;
+ }
#endif
for (i = 0; i < scns_per_block; i++) {
@@ -454,7 +458,9 @@ static bool lmb_is_removable(struct of_drconf_cell *lmb)
rc &= is_mem_section_removable(pfn, PAGES_PER_SECTION);
phys_addr += MIN_MEMORY_BLOCK_SIZE;
}
-
+ if (!rc) {
+ pr_err("is_mem_section_removable returned false \n");
+ }
return rc ? true : false;
}
@@ -465,12 +471,16 @@ static int dlpar_remove_lmb(struct of_drconf_cell
*lmb)
unsigned long block_sz;
int nid, rc;
- if (!lmb_is_removable(lmb))
+ if (!lmb_is_removable(lmb)) {
+ pr_err("dlpar_remove_lmb: lmb is not removable! \n");
return -EINVAL;
+ }
rc = dlpar_offline_lmb(lmb);
- if (rc)
+ if (rc) {
+ pr_err("dlpar_remove_lmb: offline_lmb returned not zero
\n");
return rc;
+ }
block_sz = pseries_memory_block_size();
nid = memory_add_physaddr_to_nid(lmb->base_addr);
And this is the output:
---------
[danielhb@localhost ~]$ QEMU 2.10.50 monitor - type 'help' for more
information
(qemu)
(qemu) object_add memory-backend-ram,id=ram0,size=1G
(qemu) device_add pc-dimm,id=dimm0,memdev=ram0
(qemu)
[danielhb@localhost ~]$ grep Mem /proc/meminfo
MemTotal: 5210304 kB
MemFree: 4254656 kB
MemAvailable: 4598144 kB
[danielhb@localhost ~]$ (qemu)
(qemu) device_del dimm0
(qemu) [ 136.058473] pseries-hotplug-mem: is_mem_section_removable
returned false
[ 136.058607] pseries-hotplug-mem: dlpar_remove_lmb: lmb is not removable!
[ 136.058727] pseries-hotplug-mem: Memory indexed-count-remove failed,
adding any removed LMBs
(qemu)
[danielhb@localhost ~]$ grep Mem /proc/meminfo
MemTotal: 5210304 kB
MemFree: 4253696 kB
MemAvailable: 4597184 kB
[danielhb@localhost ~]$
---------------
It appears that the hot unplug is failing because lmb_is_removable(lmb)
is returning
false inside dlpar_remove_lmb, triggering the hotplug of the LMBs again:
if (rc) {
pr_err("Memory indexed-count-remove failed, adding any
removed LMBs\n");
for (i = start_index; i < end_index; i++) {
if (!lmbs[i].reserved)
continue;
rc = dlpar_add_lmb(&lmbs[i]);
if (rc)
pr_err("Failed to add LMB, drc index %x\n",
be32_to_cpu(lmbs[i].drc_index));
lmbs[i].reserved = 0;
}
I am not aware of anything that I can do from QEMU side to fix this. Can
anyone take a look or provide
guidance? Am I missing something in my tests?
Thanks,
Daniel
