Re: [PATCH 4/7] alpha: provide ioread64 and iowrite64 implementations
> +#define iowrite64be(v,p) iowrite32(cpu_to_be64(v), (p)) Logan, thanks for taking this cleanup on. I think this should be iowrite64 not iowrite32? Stephen
lpar issue for ZONE_DEVICE p2pmem in 4.14-rc
Hi All I am hoping someone can help shed some light on an issue I am seeing with my attempt to add p2pmem [1] to the ppc64el kernel. p2pmem is a (currently) out-of-tree patchset that allows one to add device memory with struct page backing into the Linux kernel. It leverages MEMORY_HOTPLUG and ZONE_DEVICE which were added to ppc64 in 4.14 so I thought it would be fun to try it out. We constructed a patchset based off 4.14-rc1 [1] and build a kernel based off the pseries defconfig and ran this on upstream qemu-system-ppc64. The exact command to run QEMU was: qemu-system-ppc64 \ -machine pseries \ -cpu power8 \ -smp 1 -m 2048 \ -kernel ~/kernel/linux-ppc64el/vmlinux \ -append "nvme.use_cmb=24 console=hvc root=/dev/sda rootwait rw" \ -serial mon:stdio -drive file=image-ppc64el.img,if=scsi,format=raw,index=0 \ -nographic \ -drive file=../images/nvme.qcow2,if=none,id=nvme1,snapshot=on \ -device nvme,drive=nvme1,serial=nvme1 \ -drive file=../images/nvme2.qcow2,if=none,id=nvme2,snapshot=on \ -device nvme,drive=nvme2,serial=nvme2,cmb_size_mb=64 This resulted in the following extract from dmesg when registering the p2pmem associated with one of the NVMe SSDs. [3.508497] nvme :00:03.0: enabling device (0100 -> 0102) [3.510743] nvme :00:03.0: Using 64-bit direct DMA at offset 800 [3.535706] p2pmem p2pmem0: registered [3.537780] lpar: Attempting to resize HPT to shift 21 [3.539251] Unable to resize hash page table to target order 21: -1 [3.541079] Unable to create mapping for hot added memory 0xc0002100..0xc00021000400: -2 [3.550407] p2pmem p2pmem0: unregistered I am not that familiar with the pseries architecture so I was hoping for some guidance concerning the lpar error message. If any ppc64 coders fancy having a go at this the kernel code is at [1] and I’d be happy to provide the .config used if needed. Just let me know. Cheers Stephen [1] https://github.com/sbates130272/linux-p2pmem
Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc
> I am guessing that the hotplug of ZONE_DEVICE memory was done > incorrectly as it lead to HPT resizing (the system thinking this is > normal memory). Ideally one would expect that the driver would online > ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using > devm_memremap_pages() path to add these pages? > Thanks for the response Balbir. Yes we use devm_memremap_pages() to add these pages and it does call arch_add_memory(). We do have an alternate set of patches which still calls devm_memremap_pages() but can take a flag to indicate the memory being added is io memory and uses io_remap() rather than arch_add_memory() for that type of memory [1]. Would that be a better approach for this arch? I can try and apply this patch but __add_pages() has gone through some changes recently so it will take me a few days to get to that. Stephen [1] https://github.com/sbates130272/linux-p2pmem/commit/ac735871fcd2c63bd33c814aa3941ca3ef53b636
Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc
> [3.537780] lpar: Attempting to resize HPT to shift 21 > [3.539251] Unable to resize hash page table to target order 21: -1 > [3.541079] Unable to create mapping for hot added memory > 0xc0002100..0xc00021000400: -2 > For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls? Balbir do you have any suggestions as to how to test for this support? Note I am running this on my x86_64 host so there is no virtualization hardware in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.10.0-1026-gd8f932c-dirty)). > For create mapping failures, the rc is -ENOENT. Can you help debug this > further? We could do hcall tracing or enable debugging. Sure I can help debug. My original email also had all you needed to recreate this issue so that’s an option too? Stephen
Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc
Oliver >The alternative approach you mentioned that uses ioremap() should work >fine though. Great, thanks for this useful information Oliver. I am working on rebasing the ioremap() patch on 4.14 and will let you know how it goes. > Also, Alexy (+cc) said he was interested in trying this on some real > hardware. Is there a test suite for p2pmem floating around that he can > use? Excellent! I typically use p2pmem-test [1]. You will need at least one NVMe SSD and one p2pmem capable device and you *might* need to place those two devices behind a PCIe switch and configure ACS [2-3] depending on how the CPU RC treats peer to peer PCI TLPs. I am happy to help Alex set this all up so reach out either publically or privately if you like. I just added a (hopefully) decent README to help get people started. Stephen [1] https://github.com/sbates130272/p2pmem-test [2] https://www.supermicro.com/support/faqs/faq.cfm?faq=20732 [3] https://lkml.org/lkml/2017/10/26/678
