In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have:
1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but whose mmap_prepare callback clears vma->vm_ops to satisfy vma_is_anonymous(), which results in VMAs that were mmap()'d referencing a file and whose VMAs reference a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series focuses upon issue 4. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the virtual page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new virtual page offset property of VMAs to allow us to map anonymous folios at their virtual page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because firstly the approach first looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the virtual page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd:(i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability - for shared file-backed mappings, we treat the anonymous page offset as equal to the file-backed one (via vma_start_anon_pgoff()), so merge behaviour remains the same there. The only impact is on MAP_PRIVATE-mapped file-backed mappings, which must now match on virtual page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). We also address the outlier case of MAP_PRIVATE-/dev/zero mappings which have file page offset but satisfy vma_is_anonymous() by making them truly anonymous. This is low-risk as for anything meaningful these mappings behave anonymously, but it is very beneficial to do so as we eliminate an odd corner case, and those are notorious for attracting subtle bugs. Signed-off-by: Lorenzo Stoakes (ARM) <[email protected]> --- v1: - Rebased onto mm-new. - Dependent series heavily reviewed and looks highly likely to land, so un-RFC. - Added explicit check for /dev/zero and removed ability for arbitrary mmap/mmap_prepare hooks to make themselves anonymous. - Made MAP_PRIVATE-/dev/zero mappings truly anonymous. - Added MAP_PRIVATE file-backed mapping merge test to selftests. - Added a MAP_PRIVATE-/dev/zero VMA userland test to assert that the VMA really is made anonymous. - Added MAP_PRIVATE-/dev/zero merge tests to selftests. - Fixed missed virtual page index site in try_to_merge_with_ksm_page(). - Updated folio_within_range() to use virtual page offset for anon folio. This had no impact as it is only called for large folios at the moment (and MAP_PRIVATE-file backed mappings can't currently be backed by a large folio) but making the change now protects us for the future. - Fixed typos etc. RFC: https://patch.msgid.link/[email protected] To: Andrew Morton <[email protected]> To: David Hildenbrand <[email protected]> To: "Liam R. Howlett" <[email protected]> To: Vlastimil Babka <[email protected]> To: Mike Rapoport <[email protected]> To: Suren Baghdasaryan <[email protected]> To: Michal Hocko <[email protected]> To: Jann Horn <[email protected]> To: Pedro Falcato <[email protected]> To: "Matthew Wilcox (Oracle)" <[email protected]> To: Jan Kara <[email protected]> To: Miaohe Lin <[email protected]> To: Naoya Horiguchi <[email protected]> To: Rik van Riel <[email protected]> To: Harry Yoo <[email protected]> To: Lance Yang <[email protected]> To: Kees Cook <[email protected]> To: Zi Yan <[email protected]> To: Baolin Wang <[email protected]> To: Nico Pache <[email protected]> To: Ryan Roberts <[email protected]> To: Dev Jain <[email protected]> To: Barry Song <[email protected]> To: Usama Arif <[email protected]> To: Matthew Brost <[email protected]> To: Joshua Hahn <[email protected]> To: Rakie Kim <[email protected]> To: Byungchul Park <[email protected]> To: Gregory Price <[email protected]> To: Ying Huang <[email protected]> To: Alistair Popple <[email protected]> To: Peter Xu <[email protected]> To: Xu Xin <[email protected]> To: Chengming Zhou <[email protected]> To: Arnd Bergmann <[email protected]> To: Greg Kroah-Hartman <[email protected]> Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] Cc: [email protected] --- Lorenzo Stoakes (ARM) (15): mm/vma: introduce VMA virtual page offset field and add helpers mm: introduce linear_virt_page_index() mm: abstract vma_address() and introduce vma_anon_address() mm: update print_bad_page_map() to show virtual page index mm: introduce and use vma_filebacked_address() mm: propagate VMA virtual page offset on map, remap, split + merge mm/rmap: track whether the page VMA mapped walk is anonymous mm: introduce and use linear_folio_page_index() mm/rmap: use virt pgoff for MAP_PRIVATE file-backed anon folios tools/testing/vma: expand VMA merge tests to assert virt pgoff tools/testing/selftests/mm: test virtual page offset merge behaviour mm/vma: only permit MAP_PRIVATE /dev/zero to be mapped anonymous mm/vma: make MAP_PRIVATE-mapped /dev/zero mappings truly anonymous tools/testing/vma: add test to assert MAP_PRIVATE-/dev/zero is anon tools/testing/selftests/mm: add MAP_PRIVATE-/dev/zero merge tests drivers/char/mem.c | 8 +- include/linux/mm.h | 74 ++++++++++++++--- include/linux/mm_types.h | 4 + include/linux/pagemap.h | 66 +++++++++++++++ include/linux/rmap.h | 2 + mm/huge_memory.c | 3 +- mm/internal.h | 76 ++++++++++++----- mm/interval_tree.c | 4 +- mm/ksm.c | 7 +- mm/memory-failure.c | 4 +- mm/memory.c | 8 +- mm/migrate.c | 6 +- mm/mremap.c | 6 +- mm/page_vma_mapped.c | 6 +- mm/rmap.c | 23 +++--- mm/userfaultfd.c | 6 +- mm/vma.c | 116 ++++++++++++++++++++------ mm/vma.h | 144 +++++++++++++++++++++++++-------- mm/vma_exec.c | 2 +- mm/vma_init.c | 1 + mm/vma_internal.h | 1 + tools/testing/selftests/mm/merge.c | 161 +++++++++++++++++++++++++++++++++++++ tools/testing/vma/include/dup.h | 77 ++++++++++++++++++ tools/testing/vma/shared.c | 3 +- tools/testing/vma/tests/merge.c | 22 ++++- tools/testing/vma/tests/mmap.c | 49 +++++++++++ tools/testing/vma/tests/vma.c | 4 +- tools/testing/vma/vma_internal.h | 1 + 28 files changed, 764 insertions(+), 120 deletions(-) --- base-commit: 5093dba1014c1d7f7e247fd118f0fa8f22136046 change-id: 20260711-b4-scalable-cow-virt-pgoff-a0cc0eb14bc6 Cheers, -- Lorenzo Stoakes (ARM) <[email protected]>
