Hello, After playing a bit with the setrlimit calls in Linux to see how the resource RLIMIT_AS actually works, it seems to be limiting the amount of virtual memory available to the process (and not the available virtual address range).
So, I'm attempting to implement this limit in gnumach at the points memory is allocated. The patch below contains the first few steps I took in this attempt: 1. Add a `hard_limit` field, of type size_t, to the vm_map struct. I started with hard limit beacause I still need to research how soft limits work. So, for now, it's a plain rejection with ENOMEM. 2. At vm_map_setup, initialize the `hard_limit` field with the appropriate value which should be RLIM_INFINITY. In this first attempt, I hardcoded 8GB for testing. I tried a lower value, like 2GB, but some process is mapping 4GB at once during boot and it just hangs when the allocation fails. 3. Finally, enforce the limit in `vm_allocate`, `vm_map` and `vm_allocate_contiguous` by checking that current map size (`size` field) plus the requested size (`size` param) is less than the current map's `hard_limit` field. I thought of adding an RPC call that sets the `hard_limit` field which, I guess, should be located among the other task related RPCs. Also, I wanted to ask whether I covered all the allocation points or there is somewhere else where limits shall be enforced. For instance, something I still have to look at is the out-of-line data sent through a mach_msg call. Regards ---- Index: gnumach-1.8+git20240714/vm/vm_map.c =================================================================== --- gnumach-1.8+git20240714.orig/vm/vm_map.c +++ gnumach-1.8+git20240714/vm/vm_map.c @@ -198,6 +198,9 @@ void vm_map_setup( map->first_free = vm_map_to_entry(map); map->hint = vm_map_to_entry(map); map->name = NULL; + /* TODO hardcoded limit for testing purposes, rather use RLIM_INFINITY */ + /* TODO add RPC to update this limit */ + map->hard_limit = 8l * 1024l * 1024l * 1024l; vm_map_lock_init(map); simple_lock_init(&map->ref_lock); simple_lock_init(&map->hint_lock); Index: gnumach-1.8+git20240714/vm/vm_map.h =================================================================== --- gnumach-1.8+git20240714.orig/vm/vm_map.h +++ gnumach-1.8+git20240714/vm/vm_map.h @@ -198,6 +198,10 @@ struct vm_map { unsigned int timestamp; /* Version number */ const char *name; /* Associated name */ + + /* TODO only hard limits are enforced */ + /* TODO does getting rlim_t here make sense? */ + vm_size_t hard_limit; /* hard limit as set by RLIMIT_AS */ }; #define vm_map_to_entry(map) ((struct vm_map_entry *) &(map)->hdr.links) Index: gnumach-1.8+git20240714/vm/vm_user.c =================================================================== --- gnumach-1.8+git20240714.orig/vm/vm_user.c +++ gnumach-1.8+git20240714/vm/vm_user.c @@ -81,6 +81,12 @@ kern_return_t vm_allocate( *addr = trunc_page(*addr); size = round_page(size); + if (map->size + size > map->hard_limit) + { + printf("map size: %lu, requested size: %lu, hard limit: %lu\n", map->size, size, map->hard_limit); + return(KERN_NO_SPACE); + } + result = vm_map_enter( map, addr, @@ -348,6 +354,13 @@ kern_return_t vm_map( *address = trunc_page(*address); size = round_page(size); + if (target_map->size + size > target_map->hard_limit) + { + printf("map size: %lu, requested size: %lu, hard limit: %lu\n", target_map->size, size, target_map->hard_limit); + return(KERN_NO_SPACE); + } + + if (!IP_VALID(memory_object)) { object = VM_OBJECT_NULL; offset = 0; @@ -631,6 +644,12 @@ kern_return_t vm_allocate_contiguous( if (size == 0) return KERN_INVALID_ARGUMENT; + if (map->size + size > map->hard_limit) + { + printf("map size: %lu, requested size: %lu, hard limit: %lu\n", map->size, size, map->hard_limit); + return(KERN_NO_SPACE); + } + object = vm_object_allocate(size); if (object == NULL)