__vmap_pages_range_noflush() and friends can install some PTEs before failing and leave them mapped, and the kernel callers do not agree on who cleans them up. pcpu_map_pages() and kmsan_ioremap_page_range() roll back what they mapped before the failure, while vm_module_tags_populate() and the __GFP_NOFAIL retry loop in __vmalloc_area_node() rely on the mapping functions cleaning up and do not call anything like vunmap_range() themselves. When the same range is mapped again, the attempt hits the leftovers and fails with a warning in vmap_pages_pte_range(). A leftover huge pmd is worse. During the retry, vmap_try_huge_pmd() calls pmd_free_pte_page() on it, and the x86 implementation cannot tell a leaf entry from a page table and frees the mapped page.
After discussing with Suren and Ulad [1], we move the rollback into the mapping functions. A failed vmap leaves nothing mapped, and the callers no longer clean up after it. Patch 1 gives __vunmap_range_noflush() a can_resched argument, the failure paths can be reached with a non-blocking gfp mask and the undo must not schedule there. Patch 2 is the KMSAN part, the shadow and the origin of a range are mapped by two separate calls, so the rollback there has to cover both. Patches 3 and 4 reject an unaligned range at the entries of vmap_page_range() and __vunmap_range_noflush(). Patch 5 only flushes the cache once the mapping succeeded. Patches 6 to 9 add the rollback to the mapping functions, one entry per patch, vmap_page_range(), __vmap_pages_range_noflush(), the KMSAN metadata and the batched vmap() path. Patches 10 and 11 drop the powerpc and percpu cleanups that are not needed anymore. Changes since v4: - new patch 1, the unmap paths can run under a non-blocking gfp mask and must skip cond_resched(); it carries a Fixes: tag so it travels to stable together with patch 2 - drop v4 patch 2, the KMSAN-disabled vunmap skip. !kmsan_enabled does not mean no metadata was installed, KMSAN_WARN_ON() can clear the flag at runtime, on Suren's review - v4 patch 3 now warns and returns -EINVAL at vmap_page_range(), instead of BUG_ON()ing inside vmap_range_noflush() - drop v4 patch 4, the walk check. No in-tree caller passes an unaligned range to vmap_pages_range_noflush_walk(), the vmap allocator guarantees the alignment for vmalloc()/vmap()/ vm_map_ram() - v4 patch 5 warns and bails out instead of BUG_ON(), following Ulad's review of v4 - v4 patch 7 is split into patches 6 to 9, one mapping entry per patch. The batched vmap() path is covered now too, on Suren's request [1] https://lore.kernel.org/all/[email protected]/ v4 link: https://lore.kernel.org/all/[email protected]/ v3 link: https://lore.kernel.org/all/[email protected]/ v2 link: https://lore.kernel.org/all/[email protected]/ v1 link: https://lore.kernel.org/all/[email protected]/ Hao Ge (11): mm/vmalloc: skip cond_resched() in the vmap unmap paths when atomic mm/kmsan: undo the shadow mapping when the origin mapping fails mm/vmalloc: reject an unaligned range in vmap_page_range() mm/vmalloc: reject an unaligned range in __vunmap_range_noflush() mm/vmalloc: only flush the cache after the vmap succeeded mm/vmalloc: undo what vmap_page_range() mapped on failure mm/vmalloc: undo what __vmap_pages_range_noflush() mapped on failure mm/vmalloc: drop the KMSAN metadata in vmap_pages_range_noflush() on failure mm/vmalloc: undo what vmap_pages_range_batched() mapped on failure powerpc: drop redundant unmaps of failed vmap mappings mm/percpu: stop unmapping the CPU that failed to map arch/powerpc/kernel/isa-bridge.c | 5 +- arch/powerpc/kernel/pci_64.c | 4 +- include/linux/kmsan.h | 8 +- mm/kmsan/hooks.c | 18 ++-- mm/kmsan/shadow.c | 9 +- mm/percpu-vm.c | 5 +- mm/vmalloc.c | 154 +++++++++++++++++++++++-------- mm/vmalloc.h | 6 +- 8 files changed, 151 insertions(+), 58 deletions(-) -- 2.25.1
