Hi Rémy. We often fight with vm.max_map_count in the runtime, sadly. Most likely this comes from the way the runtime interacts with Linux's transparent huge page support. When we scavenge (release to the OS) only part of a huge page, we tell the OS not to turn that huge page frame back into a huge page since that would just make that memory used again. Unfortunately, each time we do this counts as a separate "mapping" just to track that one flag. These "mappings" are always at least 2MB, but you have a large enough virtual address space to reach the max_map_count even then. You can see this in /proc/PID/smaps, which should list mostly contiguous neighboring regions that differ only in a single "VmFlags" bit.
We did make memory scavenging more aggressive in Go 1.12 (+Michael Knyszek <mknys...@google.com>), though I would have expected it to converge to roughly the same "huge page flag fragmentation" as before over the course of five to ten minutes. Is your application's virtual memory footprint the same between 1.11 and 1.12 or do that grow? You could try disabling the huge page flag manipulation to confirm and/or fix this. In $GOROOT/src/runtime/internal/sys/arch_amd64.go (or whichever GOARCH is appropriate), set HugePageSize to 0. Though there's a danger that Linux's transparent huge pages could blow up your application's resident set size if you do that. On Tue, Apr 2, 2019 at 3:49 AM Rémy Oudompheng <remyoudomph...@gmail.com> wrote: > Hello, > > In a large heap program I am working on, I noticed a peculiar change in > the way virtual memory is reserved by the runtime : with comparable heap > size (about 150GB) and virtual memory size (growing to 400-500GB probably > due to a kind of fragmentation), the number of distinct memory mappings has > apparently increased between Go 1.11 and Go 1.12 reaching the system limit > (Linux setting vm.max_map_count). > > Is it something that has been experienced by someone else ? I don't > believe this classifies as a bug, but I was a bit surprised (especially as > I wasn't aware of that system limit). > > Rémy > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
