wp_huge_pmd() splits the PMD when COW or write-notify must be handled at PTE level (e.g., shared/file VMAs, userfaultfd). It then returns VM_FAULT_FALLBACK so the fault handler retries at PTE granularity. If the split fails, the PMD is still huge. The PTE fault path cannot handle a huge PMD entry. Return VM_FAULT_OOM on split failure, which signals the fault handler to invoke the OOM killer or return -ENOMEM to userspace. This is similar to what __handle_mm_fault would do if p4d_alloc or pud_alloc fails.
Signed-off-by: Usama Arif <[email protected]> --- mm/memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7ba1221c63792..51d2717e3f1b4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6161,8 +6161,13 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) } split: - /* COW or write-notify handled on pte level: split pmd. */ - __split_huge_pmd(vma, vmf->pmd, vmf->address, false); + /* + * COW or write-notify handled on pte level: split pmd. + * If split fails, the PMD is still huge so falling back + * to PTE handling would be incorrect. + */ + if (__split_huge_pmd(vma, vmf->pmd, vmf->address, false)) + return VM_FAULT_OOM; return VM_FAULT_FALLBACK; } -- 2.47.3
