hmm_range_fault() requires the caller to hold the mmap read lock for the duration of the call. This is incompatible with mappings whose fault handler may release the mmap lock, notably userfaultfd-managed regions, where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED after dropping the lock. Drivers that need to populate device page tables for such mappings have no way to do so today.
Add hmm_range_fault_unlocked() for callers that do not need to hold mmap_lock across any work outside the HMM fault itself. The new helper takes mmap_read_lock() internally, calls the common HMM fault implementation, and releases the lock before returning if it is still held. When handle_mm_fault() drops mmap_lock, hmm_range_fault_unlocked() returns -EBUSY. The caller must then restart its HMM walk with a fresh mmu_interval_read_begin() sequence. If a fatal signal is pending after the lock has been dropped, -EINTR is returned instead. This gives callers a simple restart contract while allowing fault handlers that must drop mmap_lock, including userfaultfd-backed mappings, to make progress. The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked(). The existing hmm_range_fault() path still passes no locked state, does not allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper preserving the existing API contract for current callers. The previous refactor that moved page fault handling out of the page-table walk callbacks is what makes this change small. Faults now run after walk_page_range() has unwound, with only mmap_lock held, so dropping it does not interact with the walker's pte spinlock or hugetlb_vma_lock. Hugetlb regions therefore participate in the unlocked path uniformly with PTE- and PMD-level mappings; no special case is required. Documentation/mm/hmm.rst is updated with a description of the new API and the recommended caller pattern. Signed-off-by: Stanislav Kinsburskii <[email protected]> --- Documentation/mm/hmm.rst | 59 +++++++++++++++++++++ include/linux/hmm.h | 1 mm/hmm.c | 129 ++++++++++++++++++++++++++++++++++++---------- 3 files changed, 160 insertions(+), 29 deletions(-) diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst index 7d61b7a8b65b..844343712abd 100644 --- a/Documentation/mm/hmm.rst +++ b/Documentation/mm/hmm.rst @@ -208,6 +208,65 @@ invalidate() callback. That lock must be held before calling mmu_interval_read_retry() to avoid any race with a concurrent CPU page table update. +Dropping the mmap lock during page faults +========================================= + +Some VMAs have fault handlers that need to release the mmap lock while +servicing a fault (for example, regions managed by ``userfaultfd``). +``hmm_range_fault()`` cannot be used on such mappings because it must hold the +mmap lock for the duration of the call. Drivers that need to support them +should call:: + + int hmm_range_fault_unlocked(struct hmm_range *range); + +The caller must not hold ``mmap_read_lock`` before the call. +``hmm_range_fault_unlocked()`` takes the mmap read lock internally and allows +``handle_mm_fault()`` to drop it during fault handling. If the mmap lock is +dropped, the function returns ``-EBUSY``. The caller must then restart the +walk from ``range->start`` with a fresh notifier sequence. ``-EINTR`` is +returned if a fatal signal is pending during retry handling. + +A typical caller looks like this:: + + int driver_populate_range_unlocked(...) + { + struct hmm_range range; + ... + + range.notifier = &interval_sub; + range.start = ...; + range.end = ...; + range.hmm_pfns = ...; + + if (!mmget_not_zero(interval_sub.mm)) + return -EFAULT; + + again: + range.notifier_seq = mmu_interval_read_begin(&interval_sub); + ret = hmm_range_fault_unlocked(&range); + if (ret) { + if (ret == -EBUSY) + goto again; + goto out_put; + } + + take_lock(driver->update); + if (mmu_interval_read_retry(&interval_sub, range.notifier_seq)) { + release_lock(driver->update); + goto again; + } + + /* Use pfns array content to update device page table, + * under the update lock */ + + release_lock(driver->update); + ret = 0; + + out_put: + mmput(interval_sub.mm); + return ret; + } + Leverage default_flags and pfn_flags_mask ========================================= diff --git a/include/linux/hmm.h b/include/linux/hmm.h index db75ffc949a7..1b584e5a2900 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -123,6 +123,7 @@ struct hmm_range { * Please see Documentation/mm/hmm.rst for how to use the range API. */ int hmm_range_fault(struct hmm_range *range); +int hmm_range_fault_unlocked(struct hmm_range *range); /* * HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range diff --git a/mm/hmm.c b/mm/hmm.c index 2129b1ee4c35..316116994209 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -32,6 +32,7 @@ struct hmm_vma_walk { struct hmm_range *range; + int *locked; unsigned long last; unsigned long end; unsigned int required_fault; @@ -44,6 +45,14 @@ struct hmm_vma_walk { */ #define HMM_FAULT_PENDING -EAGAIN +/* + * Internal sentinel returned by hmm_do_fault() when handle_mm_fault() + * completes a page fault with the mmap lock dropped. hmm_do_fault() sets + * *locked = 0; the outer loop consumes the sentinel and never propagates it + * to the caller. + */ +#define HMM_FAULT_UNLOCKED -ENOLCK + enum { HMM_NEED_FAULT = 1 << 0, HMM_NEED_WRITE_FAULT = 1 << 1, @@ -73,9 +82,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, * * Called by the walk callbacks when they discover that part of the range * needs a page fault. The callback records what to fault and returns - * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of - * walk_page_range() and invokes handle_mm_fault() from a context where no - * page-table or hugetlb_vma_lock is held. + * HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops + * back out of walk_page_range() and invokes handle_mm_fault() from a context + * where no page-table or hugetlb_vma_lock is held. */ static int hmm_record_fault(unsigned long addr, unsigned long end, unsigned int required_fault, @@ -624,7 +633,7 @@ static const struct mm_walk_ops hmm_walk_ops = { /* * hmm_do_fault - fault in a range recorded by a walk callback * - * Called from the outer loop in hmm_range_fault() after a callback + * Called from the outer loop in hmm_range_fault_locked() after a callback * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock; * the page-table spinlock and any hugetlb_vma_lock acquired by the walk * framework have already been released by the unwind. @@ -641,6 +650,9 @@ static int hmm_do_fault(struct mm_struct *mm, unsigned int fault_flags = FAULT_FLAG_REMOTE; struct vm_area_struct *vma; + if (hmm_vma_walk->locked) + fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; + vma = vma_lookup(mm, addr); if (!vma) return -EFAULT; @@ -651,37 +663,33 @@ static int hmm_do_fault(struct mm_struct *mm, fault_flags |= FAULT_FLAG_WRITE; } - for (; addr < end; addr += PAGE_SIZE) - if (handle_mm_fault(vma, addr, fault_flags, NULL) & - VM_FAULT_ERROR) - return -EFAULT; + for (; addr < end; addr += PAGE_SIZE) { + vm_fault_t ret; + + ret = handle_mm_fault(vma, addr, fault_flags, NULL); + + if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) { + *hmm_vma_walk->locked = 0; + return HMM_FAULT_UNLOCKED; + } + + if (ret & VM_FAULT_ERROR) { + int err = vm_fault_to_errno(ret, 0); + + if (err) + return err; + BUG(); + } + } return -EBUSY; } -/** - * hmm_range_fault - try to fault some address in a virtual address range - * @range: argument structure - * - * Returns 0 on success or one of the following error codes: - * - * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma - * (e.g., device file vma). - * -ENOMEM: Out of memory. - * -EPERM: Invalid permission (e.g., asking for write and range is read - * only). - * -EBUSY: The range has been invalidated and the caller needs to wait for - * the invalidation to finish. - * -EFAULT: A page was requested to be valid and could not be made valid - * ie it has no backing VMA or it is illegal to access - * - * This is similar to get_user_pages(), except that it can read the page tables - * without mutating them (ie causing faults). - */ -int hmm_range_fault(struct hmm_range *range) +static int hmm_range_fault_locked(struct hmm_range *range, int *locked) { struct hmm_vma_walk hmm_vma_walk = { .range = range, + .locked = locked, .last = range->start, }; struct mm_struct *mm = range->notifier->mm; @@ -704,8 +712,14 @@ int hmm_range_fault(struct hmm_range *range) * returns -EBUSY so the loop re-walks and picks up the * now-present entries. */ - if (ret == HMM_FAULT_PENDING) + if (ret == HMM_FAULT_PENDING) { ret = hmm_do_fault(mm, &hmm_vma_walk); + if (ret == HMM_FAULT_UNLOCKED) { + if (fatal_signal_pending(current)) + return -EINTR; + return -EBUSY; + } + } /* * When -EBUSY is returned the loop restarts with * hmm_vma_walk.last set to an address that has not been stored @@ -715,8 +729,65 @@ int hmm_range_fault(struct hmm_range *range) } while (ret == -EBUSY); return ret; } + +/** + * hmm_range_fault - try to fault some address in a virtual address range + * @range: argument structure + * + * Returns 0 on success or one of the following error codes: + * + * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma + * (e.g., device file vma). + * -ENOMEM: Out of memory. + * -EPERM: Invalid permission (e.g., asking for write and range is read + * only). + * -EBUSY: The range has been invalidated and the caller needs to wait for + * the invalidation to finish. + * -EFAULT: A page was requested to be valid and could not be made valid + * ie it has no backing VMA or it is illegal to access + * + * This is similar to get_user_pages(), except that it can read the page tables + * without mutating them (ie causing faults). + * + * The mmap lock must be held by the caller and will remain held on return. + * For a variant that allows the mmap lock to be dropped during faults (e.g., + * for userfaultfd support), see hmm_range_fault_unlocked(). + */ +int hmm_range_fault(struct hmm_range *range) +{ + return hmm_range_fault_locked(range, NULL); +} EXPORT_SYMBOL(hmm_range_fault); +/** + * hmm_range_fault_unlocked - fault in a range, possibly dropping the mmap lock + * @range: argument structure + * + * This is similar to hmm_range_fault(), except the caller must not hold the + * mmap lock. The function takes the mmap read lock internally and allows + * handle_mm_fault() to drop it during faults. If the lock is dropped, the + * function returns -EBUSY and the caller must restart the walk with a fresh + * mmu_interval_read_begin() sequence. + * + * Returns 0 on success or one of the error codes documented for + * hmm_range_fault(). -EINTR is returned if a fatal signal is pending during + * retry handling. + */ +int hmm_range_fault_unlocked(struct hmm_range *range) +{ + struct mm_struct *mm = range->notifier->mm; + int locked = 1; + int ret; + + mmap_read_lock(mm); + ret = hmm_range_fault_locked(range, &locked); + if (locked) + mmap_read_unlock(mm); + + return ret; +} +EXPORT_SYMBOL(hmm_range_fault_unlocked); + /** * hmm_dma_map_alloc - Allocate HMM map structure * @dev: device to allocate structure for
