From: David Woodhouse <[email protected]> A minimal in-tree implementation of the guest_memfd ops interface, useful both as reference code and as the substrate for the selftests that follow.
Features: - Backs a slot from a fixed page-less physical range specified by the module parameters addr= and len=, or falls back to a CMA-allocated contiguous region when neither is given. The addr=/len= mode lets a reserved DRAM carve-out (mem=... or an explicit e820/DT reservation) become KVM slot backing without any struct page. - Per-fd mmap capability via GMEM_PROVIDER_FLAG_MMAP_CAPABLE at GMEM_PROVIDER_SETUP time. An mmap-capable fd is bound as a gmem-only slot (KVM_MEMSLOT_GMEM_ONLY) so the host mmap and the guest fault path both route through this implementation. The mmap-capable fd is refused for hardware-encrypted VM types (SEV, SEV-ES, SEV-SNP, TDX) where a host view onto private-marked pages can only fault; the sample enforces this at bind time. KVM_X86_SW_PROTECTED_VM has no hardware encryption and is allowed. - Page revocation via GMEM_PROVIDER_SET_PRESENT. Marking a range absent tracks it in a bitmap, invokes kvm_gmem_invalidate_range() to zap the guest NPT/EPT, and fans dma_buf_invalidate_mappings() out to any exported dma-buf so iommufd tears down its mapping. Subsequent get_pfn() on the range returns -EFAULT until it is restored with present=1. Models overcommit-style reclaim without needing the platform 's real reclaim machinery. - A dma-buf exporter (GMEM_PROVIDER_GET_DMABUF) that yields an fd suitable for IOMMU_IOAS_MAP_FILE. The exporter mirrors the shape of drivers/vfio/pci/vfio_pci_dmabuf.c: dma_buf_attach_revocable, ->map_dma_buf via dma_buf_phys_vec_to_sgt, a single-range phys_vec, and the "well-known symbol" hook for iommufd's private-interconnect phys-map lookup. The sample is layered on top of the shared kvm_gmem_fops: its fd is created with those fops and file->private_data points at a kvm_gmem_backing embedded at the head of its per-instance state. Release / mmap / ioctl for the fd all run through the ops table, so the sample owns no file_operations of its own. Exposing kvm_gmem_fops for module use required promoting its export from EXPORT_SYMBOL_FOR_KVM_INTERNAL to EXPORT_SYMBOL_GPL and adding it to the KVM external-export allowlist. SEV-SNP bind takes a psmash + rmp_make_shared sweep over the range so a new SNP VM can transition its pages private and re-encrypt them -- harmless on non-SNP hosts, where the calls return -ENODEV. Based on a proof-of-concept by Connor Williamson <[email protected]>. Signed-off-by: David Woodhouse (Kiro) <[email protected]> --- arch/x86/kvm/Makefile | 3 +- samples/Kconfig | 18 + samples/Makefile | 1 + samples/kvm/Makefile | 2 + samples/kvm/gmem_provider.c | 651 ++++++++++++++++++++++++++++++++++++ samples/kvm/gmem_provider.h | 53 +++ virt/kvm/guest_memfd.c | 2 +- 7 files changed, 728 insertions(+), 2 deletions(-) create mode 100644 samples/kvm/Makefile create mode 100644 samples/kvm/gmem_provider.c create mode 100644 samples/kvm/gmem_provider.h diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index cb963d6ecaab..535fb8cc259f 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -64,7 +64,8 @@ exports_grep_trailer := --include='*.[ch]' -nrw $(srctree)/virt/kvm $(srctree)/a -e kvm_get_kvm \ -e kvm_get_kvm_safe \ -e kvm_put_kvm \ - -e kvm_gmem_invalidate_range + -e kvm_gmem_invalidate_range \ + -e kvm_gmem_fops # Force grep to emit a goofy group separator that can in turn be replaced with # the above newline macro (newlines in Make are a nightmare). Note, grep only diff --git a/samples/Kconfig b/samples/Kconfig index a75e8e78330d..3482659638b5 100644 --- a/samples/Kconfig +++ b/samples/Kconfig @@ -326,6 +326,24 @@ source "samples/rust/Kconfig" source "samples/damon/Kconfig" +config SAMPLE_KVM_GMEM_PROVIDER + tristate "Build sample guest_memfd provider -- loadable module only" + depends on KVM_GUEST_MEMFD && X86_64 && m + help + This builds a sample external guest_memfd provider module with two + backing modes. Loaded with addr=/len= it backs guest memory with a + fixed physical range that has no struct page -- for example memory + carved out of the kernel with mem= on the command line. Without + those params it falls back to a physically contiguous, page-backed + region from alloc_contig_pages(), sized by the setup ioctl (easier to + run, no boot param). + + It demonstrates the guest_memfd provider ABI including 2M and 1G + guest mappings and, on SEV-SNP hosts, the RMP reset that lets a + page-less range be re-bound to a new VM across a live update. + + If unsure, say N. + endif # SAMPLES config HAVE_SAMPLE_FTRACE_DIRECT diff --git a/samples/Makefile b/samples/Makefile index 07641e177bd8..e85397e5e34f 100644 --- a/samples/Makefile +++ b/samples/Makefile @@ -37,6 +37,7 @@ obj-$(CONFIG_SAMPLE_TPS6594_PFSM) += pfsm/ subdir-$(CONFIG_SAMPLE_WATCHDOG) += watchdog subdir-$(CONFIG_SAMPLE_WATCH_QUEUE) += watch_queue obj-$(CONFIG_SAMPLE_KMEMLEAK) += kmemleak/ +obj-$(CONFIG_SAMPLE_KVM_GMEM_PROVIDER) += kvm/ obj-$(CONFIG_SAMPLE_CORESIGHT_SYSCFG) += coresight/ obj-$(CONFIG_SAMPLE_FPROBE) += fprobe/ obj-$(CONFIG_SAMPLES_RUST) += rust/ diff --git a/samples/kvm/Makefile b/samples/kvm/Makefile new file mode 100644 index 000000000000..dcad6e53ea78 --- /dev/null +++ b/samples/kvm/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_SAMPLE_KVM_GMEM_PROVIDER) += gmem_provider.o diff --git a/samples/kvm/gmem_provider.c b/samples/kvm/gmem_provider.c new file mode 100644 index 000000000000..e30580fecd16 --- /dev/null +++ b/samples/kvm/gmem_provider.c @@ -0,0 +1,651 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * gmem_provider - sample external guest_memfd provider. + * + * Demonstrates the KVM guest_memfd provider ABI with two backing modes: + * + * - External (page-less): loaded with addr=/len=, backs guest memory with a + * fixed physical range that has no struct page -- e.g. memory carved out of + * the kernel with mem= on the command line. This is the case the provider + * ABI exists for; get_pfn() returns bare PFNs KVM treats as non-refcounted. + * + * - CMA fallback (page-backed): when addr=/len= are not given, allocates a + * physically contiguous region via alloc_contig_pages() sized by the setup + * ioctl. Easier to run (no mem= boot param), and still exercises 2M/1G + * mappings. Not preserved across kexec/live update. + * + * On SEV-SNP hosts, bind() resets the range's RMP entries to 4K shared so a + * (new) SNP VM can re-encrypt it, which is what allows re-binding the range to + * a fresh VM across a live update. + * + * Usage: + * # page-less external range: + * insmod gmem_provider.ko addr=0x5D40000000 len=0x1000000 + * # or CMA fallback (no params); size comes from the ioctl + * insmod gmem_provider.ko + * fd = open("/dev/gmem_provider"); ioctl(fd, GMEM_PROVIDER_SETUP, {kvm_fd, size}); + * pass the returned fd + KVM_MEM_GUEST_MEMFD to KVM_SET_USER_MEMORY_REGION2. + */ + +#include <linux/anon_inodes.h> +#include <linux/bitmap.h> +#include <linux/dma-buf.h> +#include <linux/dma-buf-mapping.h> +#include <linux/dma-resv.h> +#include <linux/file.h> +#include <linux/fs.h> +#include <linux/gfp.h> +#include <linux/highmem.h> +#include <linux/io.h> +#include <linux/kvm_host.h> +#include <linux/miscdevice.h> +#include <linux/mm.h> +#include <linux/module.h> +#include <linux/pgtable.h> +#include <linux/slab.h> + +#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) +#include <asm/sev.h> +#endif + +#include "gmem_provider.h" + +MODULE_IMPORT_NS("DMA_BUF"); + +static unsigned long long addr; +module_param(addr, ullong, 0444); +MODULE_PARM_DESC(addr, "physical base of an external page-less backing region (optional)"); + +static unsigned long long len; +module_param(len, ullong, 0444); +MODULE_PARM_DESC(len, "size in bytes of the external backing region (optional)"); + + +struct gmem_info { + /* + * MUST be first: file->private_data points here. is_kvm_gmem_file() + * on the KVM side proves the reinterpretation is safe. + */ + struct kvm_gmem_backing backing; + + /* Kept locally: kvm_gmem_ops does not carry a kvm pointer. */ + struct kvm *kvm; + + bool mmap_capable; /* -> KVM_MEMSLOT_GMEM_ONLY at bind */ + + unsigned long base_pfn; + unsigned long npages; + struct page *cma_pages; /* non-NULL if CMA-allocated */ + gfn_t base_gfn; /* recorded at bind, for revoke */ + pgoff_t pgoff; /* provider offset (pages) of the slot */ + unsigned long *absent; /* bitmap of currently-revoked pages */ + struct list_head dmabufs; /* struct gmem_dmabuf entries */ + struct mutex dmabufs_lock; +}; + +static struct gmem_info *to_gmem_info(struct file *file) +{ + return container_of(file->private_data, struct gmem_info, backing); +} + +/* Map a backing PFN for CPU access: page-backed via kmap, page-less via memremap. */ +static void *gmem_map_pfn(kvm_pfn_t pfn) +{ + if (pfn_valid(pfn)) + return kmap_local_pfn(pfn); + return memremap(PFN_PHYS(pfn), PAGE_SIZE, MEMREMAP_WB); +} + +static void gmem_unmap_pfn(kvm_pfn_t pfn, void *vaddr) +{ + if (!vaddr) + return; + if (pfn_valid(pfn)) + kunmap_local(vaddr); + else + memunmap(vaddr); +} + +/* Largest order KVM may map at @gfn, snapped to 4K/2M/1G. */ +static int gmem_max_order(struct gmem_info *info, gfn_t gfn, unsigned long index) +{ + unsigned long pfn = info->base_pfn + index; + unsigned long remaining = info->npages - index; + unsigned int pud_order = PUD_SHIFT - PAGE_SHIFT; + unsigned int pmd_order = PMD_SHIFT - PAGE_SHIFT; + + if (IS_ALIGNED(pfn, 1UL << pud_order) && + IS_ALIGNED(gfn, 1UL << pud_order) && + remaining >= (1UL << pud_order)) + return pud_order; + + if (IS_ALIGNED(pfn, 1UL << pmd_order) && + IS_ALIGNED(gfn, 1UL << pmd_order) && + remaining >= (1UL << pmd_order)) + return pmd_order; + + return 0; +} + +static int gmem_get_pfn(struct file *file, struct kvm *kvm, + struct kvm_memory_slot *slot, gfn_t gfn, + kvm_pfn_t *pfn, struct page **page, int *max_order) +{ + struct gmem_info *info = to_gmem_info(file); + pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff; + + if (index >= info->npages) + return -EINVAL; + + /* Revoked (absent) page: behave like not-present so the fault fails. */ + if (info->absent && test_bit(index, info->absent)) + return -EFAULT; + + *pfn = info->base_pfn + index; + if (max_order) + *max_order = gmem_max_order(info, gfn, index); + return 0; +} + +static int gmem_populate(struct file *file, struct kvm *kvm, + struct kvm_memory_slot *slot, gfn_t gfn, + kvm_pfn_t *pfn, struct page *src_page, int order) +{ + struct gmem_info *info = to_gmem_info(file); + pgoff_t index = gfn - slot->base_gfn + slot->gmem.pgoff; + + if (index >= info->npages) + return -EINVAL; + + *pfn = info->base_pfn + index; + + if (src_page) { + void *dst, *src; + + /* Map dst first: memremap() may sleep, kmap_local_page() must not. */ + dst = gmem_map_pfn(*pfn); + if (!dst) + return -ENOMEM; + src = kmap_local_page(src_page); + memcpy(dst, src, PAGE_SIZE); + kunmap_local(src); + gmem_unmap_pfn(*pfn, dst); + } + return 0; +} + +static int gmem_bind(struct file *file, struct kvm *kvm, + struct kvm_memory_slot *slot, loff_t offset) +{ + struct gmem_info *info = to_gmem_info(file); + unsigned long start = offset >> PAGE_SHIFT; + + if (offset < 0 || !PAGE_ALIGNED(offset) || + start + slot->npages > info->npages) + return -EINVAL; + + /* + * An mmap-capable backing hands the VMM a host mapping onto pages + * that a hardware-encrypted VM (SEV, SEV-ES, SEV-SNP, TDX) will mark + * private in the RMP/EPT; the mmap can only ever fault on those + * pages. Refuse rather than hand the VMM a useless (and misleading) + * shared view. SW_PROTECTED_VM has no hardware encryption and is + * fine. + */ +#ifdef CONFIG_X86 + if (info->mmap_capable && + (kvm->arch.vm_type == KVM_X86_SEV_VM || + kvm->arch.vm_type == KVM_X86_SEV_ES_VM || + kvm->arch.vm_type == KVM_X86_SNP_VM || + kvm->arch.vm_type == KVM_X86_TDX_VM)) + return -EACCES; +#endif + + /* Record the binding so the revoke ioctl can translate offset -> gfn. */ + info->base_gfn = slot->base_gfn; + info->pgoff = start; + +#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) + /* + * Reset the RMP for the range to 4K shared so a (new) SEV-SNP VM can + * transition it to private and re-encrypt it. PSMASH any 2M entries + * first. Harmless on non-SNP hosts, where these return -ENODEV. + */ + { + unsigned long i; + + for (i = 0; i < slot->npages; i += PTRS_PER_PMD) { + unsigned long pfn = info->base_pfn + start + i; + + if (IS_ALIGNED(pfn, PTRS_PER_PMD)) + psmash(pfn); + } + for (i = 0; i < slot->npages; i++) { + unsigned long pfn = info->base_pfn + start + i; + int ret = rmp_make_shared(pfn, PG_LEVEL_4K); + + if (ret && ret != -ENODEV) + pr_info_once("gmem_provider: rmp_make_shared(0x%lx) = %d\n", + pfn, ret); + } + } +#endif + + /* Claim (or, on re-bind, transfer) VM ownership; pins the VM. */ + if (info->kvm != kvm) { + kvm_get_kvm(kvm); + if (info->kvm) + kvm_put_kvm(info->kvm); + info->kvm = kvm; + } + + /* + * Record the file on the slot (KVM's outer bind no longer does this + * for us) and mark the slot gmem-only if this backing serves host + * accesses through its own mmap. + */ + WRITE_ONCE(slot->gmem.file, file); + slot->gmem.pgoff = start; + if (info->mmap_capable) + slot->flags |= KVM_MEMSLOT_GMEM_ONLY; + + return 0; +} + +static void gmem_unbind(struct file *file, struct kvm *kvm, + struct kvm_memory_slot *slot) +{ +#if IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT) + struct gmem_info *info = to_gmem_info(file); + unsigned long start = slot->gmem.pgoff; + unsigned long i; + + for (i = 0; i < slot->npages; i++) { + unsigned long pfn = info->base_pfn + start + i; + + rmp_make_shared(pfn, PG_LEVEL_4K); + } +#endif +} + +static void gmem_release(struct file *file); +static int gmem_mmap(struct file *file, struct vm_area_struct *vma); +static long gmem_fd_ioctl(struct file *file, unsigned int cmd, unsigned long arg); + +static const struct kvm_gmem_ops gmem_ops = { + .bind = gmem_bind, + .unbind = gmem_unbind, + .get_pfn = gmem_get_pfn, + .populate = gmem_populate, + .release = gmem_release, + .mmap = gmem_mmap, + .ioctl = gmem_fd_ioctl, +}; + +static void gmem_release(struct file *file) +{ + struct gmem_info *info = to_gmem_info(file); + + if (info->kvm) + kvm_put_kvm(info->kvm); + if (info->cma_pages) + free_contig_range(info->base_pfn, info->npages); + bitmap_free(info->absent); + kfree(info); +} + +static int gmem_mmap(struct file *file, struct vm_area_struct *vma) +{ + struct gmem_info *info = to_gmem_info(file); + unsigned long npages = vma_pages(vma); + + if (vma->vm_pgoff + npages > info->npages) + return -EINVAL; + + /* Page-less backing: map raw PFNs, not folios. */ + vm_flags_set(vma, VM_PFNMAP | VM_IO | VM_DONTEXPAND | VM_DONTDUMP); + return remap_pfn_range(vma, vma->vm_start, info->base_pfn + vma->vm_pgoff, + npages << PAGE_SHIFT, vma->vm_page_prot); +} + +/* + * Dynamic dma-buf exporter over the provider's backing. + * + * Follows the same shape as drivers/vfio/pci/vfio_pci_dmabuf.c: a per-dmabuf + * priv holding a phys_vec, a revocable dynamic attach, and a "private + * interconnect" symbol iommufd looks up to fetch phys directly (instead of + * mapping through the DMA API). + * + * A revoke on the provider (SET_PRESENT present=0) fans out to every exported + * dma-buf via dma_buf_invalidate_mappings(), so iommufd (which registered a + * revocable importer) tears down the IOMMU mapping alongside KVM's NPT zap. + */ +struct gmem_dmabuf { + struct dma_buf *dmabuf; + struct gmem_info *info; + struct file *provider_file; /* holds info alive */ + struct list_head list; /* info->dmabufs */ + struct phys_vec phys; /* single contiguous range */ + struct kref kref; + struct completion comp; + bool revoked; +}; + +static int gmem_dma_buf_attach(struct dma_buf *dmabuf, + struct dma_buf_attachment *attach) +{ + struct gmem_dmabuf *priv = dmabuf->priv; + + if (!attach->peer2peer) + return -EOPNOTSUPP; + if (priv->revoked) + return -ENODEV; + if (!dma_buf_attach_revocable(attach)) + return -EOPNOTSUPP; + return 0; +} + +static void gmem_dma_buf_done(struct kref *kref) +{ + struct gmem_dmabuf *priv = container_of(kref, struct gmem_dmabuf, kref); + + complete(&priv->comp); +} + +static struct sg_table *gmem_dma_buf_map(struct dma_buf_attachment *attach, + enum dma_data_direction dir) +{ + struct gmem_dmabuf *priv = attach->dmabuf->priv; + struct sg_table *sgt; + + dma_resv_assert_held(priv->dmabuf->resv); + if (priv->revoked) + return ERR_PTR(-ENODEV); + + /* RAM, not P2P MMIO: no p2pdma_provider. */ + sgt = dma_buf_phys_vec_to_sgt(attach, NULL, &priv->phys, 1, + priv->phys.len, dir); + if (IS_ERR(sgt)) + return sgt; + + kref_get(&priv->kref); + return sgt; +} + +static void gmem_dma_buf_unmap(struct dma_buf_attachment *attach, + struct sg_table *sgt, + enum dma_data_direction dir) +{ + struct gmem_dmabuf *priv = attach->dmabuf->priv; + + dma_resv_assert_held(priv->dmabuf->resv); + dma_buf_free_sgt(attach, sgt, dir); + kref_put(&priv->kref, gmem_dma_buf_done); +} + +static void gmem_dma_buf_release(struct dma_buf *dmabuf) +{ + struct gmem_dmabuf *priv = dmabuf->priv; + + if (priv->info) { + mutex_lock(&priv->info->dmabufs_lock); + list_del_init(&priv->list); + mutex_unlock(&priv->info->dmabufs_lock); + } + if (priv->provider_file) + fput(priv->provider_file); + kfree(priv); +} + +static const struct dma_buf_ops gmem_dma_buf_ops = { + .attach = gmem_dma_buf_attach, + .map_dma_buf = gmem_dma_buf_map, + .unmap_dma_buf = gmem_dma_buf_unmap, + .release = gmem_dma_buf_release, +}; + +/* + * Private interconnect for iommufd (mirrors vfio_pci_dma_buf_iommufd_map). + * Returns the single contiguous phys range for the exported region so iommufd + * can program the IOMMU directly, bypassing the DMA API. + */ +int gmem_provider_dma_buf_iommufd_map(struct dma_buf_attachment *attach, + struct phys_vec *phys); +int gmem_provider_dma_buf_iommufd_map(struct dma_buf_attachment *attach, + struct phys_vec *phys) +{ + struct gmem_dmabuf *priv; + + dma_resv_assert_held(attach->dmabuf->resv); + if (attach->dmabuf->ops != &gmem_dma_buf_ops) + return -EOPNOTSUPP; + priv = attach->dmabuf->priv; + if (priv->revoked) + return -ENODEV; + *phys = priv->phys; + return 0; +} +EXPORT_SYMBOL_FOR_MODULES(gmem_provider_dma_buf_iommufd_map, "iommufd"); + +/* Called with info->dmabufs_lock held on the revoke path. */ +static void gmem_dma_buf_revoke_all(struct gmem_info *info) +{ + struct gmem_dmabuf *priv; + + list_for_each_entry(priv, &info->dmabufs, list) { + dma_resv_lock(priv->dmabuf->resv, NULL); + if (!priv->revoked) { + priv->revoked = true; + dma_buf_invalidate_mappings(priv->dmabuf); + } + dma_resv_unlock(priv->dmabuf->resv); + } +} + +static int gmem_provider_get_dmabuf(struct file *file) +{ + struct gmem_info *info = to_gmem_info(file); + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); + struct gmem_dmabuf *priv; + int fd; + + priv = kzalloc(sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + + priv->info = info; + priv->provider_file = get_file(file); + priv->phys.paddr = (u64)info->base_pfn << PAGE_SHIFT; + priv->phys.len = (u64)info->npages << PAGE_SHIFT; + kref_init(&priv->kref); + init_completion(&priv->comp); + INIT_LIST_HEAD(&priv->list); + + exp_info.ops = &gmem_dma_buf_ops; + exp_info.size = priv->phys.len; + exp_info.flags = O_RDWR; + exp_info.priv = priv; + + priv->dmabuf = dma_buf_export(&exp_info); + if (IS_ERR(priv->dmabuf)) { + fd = PTR_ERR(priv->dmabuf); + kfree(priv); + return fd; + } + + mutex_lock(&info->dmabufs_lock); + list_add(&priv->list, &info->dmabufs); + mutex_unlock(&info->dmabufs_lock); + + fd = dma_buf_fd(priv->dmabuf, O_CLOEXEC); + if (fd < 0) + dma_buf_put(priv->dmabuf); + return fd; +} + +static long gmem_fd_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + struct gmem_info *info = to_gmem_info(file); + struct gmem_provider_present p; + unsigned long start_index, end_index; + + if (cmd == GMEM_PROVIDER_GET_DMABUF) + return gmem_provider_get_dmabuf(file); + + if (cmd != GMEM_PROVIDER_SET_PRESENT) + return -ENOTTY; + if (copy_from_user(&p, (void __user *)arg, sizeof(p))) + return -EFAULT; + if (!p.len || !PAGE_ALIGNED(p.offset) || !PAGE_ALIGNED(p.len)) + return -EINVAL; + + start_index = p.offset >> PAGE_SHIFT; + end_index = start_index + (p.len >> PAGE_SHIFT); + if (end_index > info->npages || end_index < start_index) + return -EINVAL; + + if (p.present) { + /* Restore: next guest fault calls get_pfn() and re-maps. */ + bitmap_clear(info->absent, start_index, end_index - start_index); + } else { + /* Revoke: mark absent, then zap the guest NPT/EPT for the range. */ + bitmap_set(info->absent, start_index, end_index - start_index); + if (info->kvm) + kvm_gmem_invalidate_range(info->kvm, + info->base_gfn + start_index - info->pgoff, + info->base_gfn + end_index - info->pgoff); + + /* + * Fan out to iommufd (and any other dma-buf importer): mark the + * exported dma-buf(s) revoked and invalidate any active mappings. + * The provider stays ignorant of scratch-page policy; that lives + * in the importer. + */ + mutex_lock(&info->dmabufs_lock); + gmem_dma_buf_revoke_all(info); + mutex_unlock(&info->dmabufs_lock); + } + return 0; +} + +static long gmem_ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + struct gmem_provider_setup setup; + struct gmem_info *info; + struct file *kvm_file; + struct page *pages = NULL; + struct kvm *kvm; + unsigned long npages; + int fd, ret; + + if (cmd != GMEM_PROVIDER_SETUP) + return -ENOTTY; + + if (copy_from_user(&setup, (void __user *)arg, sizeof(setup))) + return -EFAULT; + if (setup.flags & ~GMEM_PROVIDER_FLAG_MMAP_CAPABLE) + return -EINVAL; + + kvm_file = fget(setup.kvm_fd); + if (!kvm_file) + return -EBADF; + kvm = kvm_file->private_data; + if (!kvm) { + fput(kvm_file); + return -EINVAL; + } + kvm_get_kvm(kvm); + fput(kvm_file); + + info = kzalloc(sizeof(*info), GFP_KERNEL); + if (!info) { + ret = -ENOMEM; + goto err_put_kvm; + } + + if (addr && len) { + /* External page-less range from module params. */ + info->base_pfn = addr >> PAGE_SHIFT; + info->npages = len >> PAGE_SHIFT; + } else { + /* CMA fallback: allocate a contiguous, page-backed region. */ + if (!setup.size || !PAGE_ALIGNED(setup.size)) { + ret = -EINVAL; + goto err_free_info; + } + npages = setup.size >> PAGE_SHIFT; + pages = alloc_contig_pages(npages, GFP_KERNEL, numa_node_id(), NULL); + if (!pages) { + ret = -ENOMEM; + goto err_free_info; + } + /* Provider path skips KVM's folio-clear; zero to avoid data leak. */ + memset(page_to_virt(pages), 0, (size_t)npages << PAGE_SHIFT); + info->base_pfn = page_to_pfn(pages); + info->npages = npages; + info->cma_pages = pages; + } + + info->absent = bitmap_zalloc(info->npages, GFP_KERNEL); + if (!info->absent) { + ret = -ENOMEM; + goto err_free_pages; + } + + INIT_LIST_HEAD(&info->dmabufs); + mutex_init(&info->dmabufs_lock); + + info->backing.ops = &gmem_ops; + info->kvm = kvm; + info->mmap_capable = !!(setup.flags & GMEM_PROVIDER_FLAG_MMAP_CAPABLE); + + fd = anon_inode_getfd("[gmem-provider]", &kvm_gmem_fops, + &info->backing, O_RDWR | O_CLOEXEC); + if (fd < 0) { + ret = fd; + goto err_free_pages; + } + return fd; + +err_free_pages: + if (pages) + free_contig_range(page_to_pfn(pages), npages); +err_free_info: + bitmap_free(info->absent); + kfree(info); +err_put_kvm: + kvm_put_kvm(kvm); + return ret; +} + +static const struct file_operations gmem_ctl_fops = { + .owner = THIS_MODULE, + .unlocked_ioctl = gmem_ctl_ioctl, + .compat_ioctl = gmem_ctl_ioctl, +}; + +static struct miscdevice gmem_dev = { + .minor = MISC_DYNAMIC_MINOR, + .name = "gmem_provider", + .fops = &gmem_ctl_fops, +}; + +static int __init gmem_provider_init(void) +{ + if ((addr || len) && + (!addr || !len || !PAGE_ALIGNED(addr) || !PAGE_ALIGNED(len))) { + pr_err("gmem_provider: addr= and len= must both be set and page aligned\n"); + return -EINVAL; + } + return misc_register(&gmem_dev); +} +module_init(gmem_provider_init); + +static void __exit gmem_provider_exit(void) +{ + misc_deregister(&gmem_dev); +} +module_exit(gmem_provider_exit); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("Sample guest_memfd provider (external page-less range or CMA fallback)"); diff --git a/samples/kvm/gmem_provider.h b/samples/kvm/gmem_provider.h new file mode 100644 index 000000000000..45f1b8257f60 --- /dev/null +++ b/samples/kvm/gmem_provider.h @@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _SAMPLES_KVM_GMEM_PROVIDER_H +#define _SAMPLES_KVM_GMEM_PROVIDER_H + +#include <linux/ioctl.h> +#include <linux/types.h> + +/* + * ioctl on /dev/gmem_provider: create a guest_memfd provider fd and return it + * for use with KVM_SET_USER_MEMORY_REGION2. + * + * If the module was loaded with addr=/len=, the fd is backed by that fixed + * page-less physical range and @size is ignored. Otherwise the fd is backed + * by a @size-byte physically contiguous region from alloc_contig_pages(). + */ +/* Flags for struct gmem_provider_setup.flags */ +#define GMEM_PROVIDER_FLAG_MMAP_CAPABLE (1u << 0) /* fd is mmap()-able; slot becomes gmem-only. + Refused for coco VMs (SEV-SNP/TDX). */ + +struct gmem_provider_setup { + __s32 kvm_fd; /* an open KVM VM fd */ + __u32 flags; /* GMEM_PROVIDER_FLAG_* */ + __u64 size; /* CMA fallback size in bytes, page aligned */ +}; + +#define GMEM_PROVIDER_IOCTL_BASE 'G' +#define GMEM_PROVIDER_SETUP _IOW(GMEM_PROVIDER_IOCTL_BASE, 1, struct gmem_provider_setup) + +/* + * ioctl on a provider fd (returned by SETUP): flip a byte range of the backing + * between present and absent. Revoking (present=0) marks the range absent and + * zaps the guest's NPT/EPT so the next access re-faults; get_pfn() then refuses + * the range until it is restored (present=1). Models overcommit page reclaim. + */ +struct gmem_provider_present { + __u64 offset; /* byte offset into the provider region, page aligned */ + __u64 len; /* byte length, page aligned */ + __u32 present; /* 0 = revoke (absent), 1 = restore (present) */ + __u32 pad; +}; + +#define GMEM_PROVIDER_SET_PRESENT _IOW(GMEM_PROVIDER_IOCTL_BASE, 2, struct gmem_provider_present) + +/* + * ioctl on a provider fd (returned by SETUP): export the backing region as a + * dynamic dma-buf and return an fd for it, suitable for + * IOMMU_IOAS_MAP_FILE. Revoking the region (SET_PRESENT present=0) fans out + * to the exported dma-buf via dma_buf_invalidate_mappings(), causing iommufd + * to tear down the IOMMU mapping so DMA to the reclaimed range faults. + */ +#define GMEM_PROVIDER_GET_DMABUF _IO(GMEM_PROVIDER_IOCTL_BASE, 3) + +#endif /* _SAMPLES_KVM_GMEM_PROVIDER_H */ diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 6c13742de068..2cb1b6a9d73f 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -546,7 +546,7 @@ struct file_operations kvm_gmem_fops = { .unlocked_ioctl = kvm_gmem_fops_ioctl, .compat_ioctl = kvm_gmem_fops_ioctl, }; -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gmem_fops); +EXPORT_SYMBOL_GPL(kvm_gmem_fops); static int kvm_gmem_migrate_folio(struct address_space *mapping, struct folio *dst, struct folio *src, -- 2.54.0
