As you know, QEMU can't direct it's memory allocation now, this may cause guest cross node access performance regression. And, the worse thing is that if PCI-passthrough is used, direct-attached-device uses DMA transfer between device and qemu process. All pages of the guest will be pinned by get_user_pages().
KVM_ASSIGN_PCI_DEVICE ioctl kvm_vm_ioctl_assign_device() =>kvm_assign_device() => kvm_iommu_map_memslots() => kvm_iommu_map_pages() => kvm_pin_pages() So, with direct-attached-device, all guest page's page count will be +1 and any page migration will not work. AutoNUMA won't too. So, we should set the guest nodes memory allocation policy before the pages are really mapped. According to this patch set, we are able to set guest nodes memory policy like following: -numa node,nodeid=0,mem=1024,cpus=0,mem-policy=membind,mem-hostnode=0-1 -numa node,nodeid=1,mem=1024,cpus=1,mem-policy=interleave,mem-hostnode=1 This supports "mem-policy={membind|interleave|preferred},mem-hostnode=[+|!]{all|N-N}" like format. And patch 8/10 adds a QMP command "set-mpol" to set the memory policy for every guest nodes: set-mpol nodeid=0 mem-policy=membind mem-hostnode=0-1 And patch 9/10 adds a monitor command "set-mpol" whose format like: set-mpol 0 mem-policy=membind,mem-hostnode=0-1 And with patch 10/10, we can get the current memory policy of each guest node using monitor command "info numa", for example: (qemu) info numa 2 nodes node 0 cpus: 0 node 0 size: 1024 MB node 0 mempolicy: membind=0,1 node 1 cpus: 1 node 1 size: 1024 MB node 1 mempolicy: interleave=1 V1->V2: change to use QemuOpts in numa options (Paolo) handle Error in mpol parser (Paolo) change qmp command format to mem-policy=membind,mem-hostnode=0-1 like (Paolo) V2->V3: also handle Error in cpus parser (5/10) split out common parser from cpus and hostnode parser (Bandan 6/10) V3-V4: rebase to request for comments Bandan Das (1): NUMA: Support multiple CPU ranges on -numa option Wanlong Gao (9): NUMA: Add numa_info structure to contain numa nodes info NUMA: Add Linux libnuma detection NUMA: parse guest numa nodes memory policy NUMA: handle Error in cpus, mpol and hostnode parser NUMA: split out the common range parser NUMA: set guest numa nodes memory policy NUMA: add qmp command set-mpol to set memory policy for NUMA node NUMA: add hmp command set-mpol NUMA: show host memory policy info in info numa command configure | 32 ++++++ cpus.c | 143 +++++++++++++++++++++++- hmp-commands.hx | 16 +++ hmp.c | 35 ++++++ hmp.h | 1 + hw/i386/pc.c | 4 +- hw/net/eepro100.c | 1 - include/sysemu/sysemu.h | 20 +++- monitor.c | 44 +++++++- qapi-schema.json | 15 +++ qemu-options.hx | 3 +- qmp-commands.hx | 35 ++++++ vl.c | 285 +++++++++++++++++++++++++++++++++++------------- 13 files changed, 553 insertions(+), 81 deletions(-) -- 1.8.3.1.448.gfb7dfaa