Thanks Austin, The application workload is a kind of fragmentation torture test as it involves a mixture of many long-lived small and large (>100 MB) objects, with regularly allocated short-lived small and large objects. I have tried creating a sample synthetic reproducer but did not succeed at the moment.
Regarding the max_map_count, your explanation is very clear and I apparently missed the large comment in the runtime explaining all of that. Do you expect a significant drawback between choosing 2MB or 16MB as the granularity of the huge page flag manipulation in the case of huge heaps ? Regarding the virtual memory footprint, it changed radically with Go 1.12. It basically looks like a leak and I saw it grow to more than 1TB where the actual heap total size never exceeds 180GB. Although I understand that it is easy to construct a situation where there is repeatedly no available contiguous interval of >100MB in the address space, it is a significant difference from Go 1.11 where the address space would grow to 400-500GB for a similar workload and stay flat after that, and I could not find an obvious change in the allocator explaining the phenomenon (and unfortunately my resources do not allow for an easy live comparison of both program lifetimes). Am I right saying that scavenging method or frequency does not (cannot) affect at all virtual memory footprint and dynamics ? Regards, Rémy. Le mar. 2 avr. 2019 à 16:15, Austin Clements <aus...@google.com> a écrit : > > Hi Rémy. We often fight with vm.max_map_count in the runtime, sadly. Most > likely this comes from the way the runtime interacts with Linux's transparent > huge page support. When we scavenge (release to the OS) only part of a huge > page, we tell the OS not to turn that huge page frame back into a huge page > since that would just make that memory used again. Unfortunately, each time > we do this counts as a separate "mapping" just to track that one flag. These > "mappings" are always at least 2MB, but you have a large enough virtual > address space to reach the max_map_count even then. You can see this in > /proc/PID/smaps, which should list mostly contiguous neighboring regions that > differ only in a single "VmFlags" bit. > > We did make memory scavenging more aggressive in Go 1.12 (+Michael Knyszek), > though I would have expected it to converge to roughly the same "huge page > flag fragmentation" as before over the course of five to ten minutes. Is your > application's virtual memory footprint the same between 1.11 and 1.12 or do > that grow? > > You could try disabling the huge page flag manipulation to confirm and/or fix > this. In $GOROOT/src/runtime/internal/sys/arch_amd64.go (or whichever GOARCH > is appropriate), set HugePageSize to 0. Though there's a danger that Linux's > transparent huge pages could blow up your application's resident set size if > you do that. > > On Tue, Apr 2, 2019 at 3:49 AM Rémy Oudompheng <remyoudomph...@gmail.com> > wrote: >> >> Hello, >> >> In a large heap program I am working on, I noticed a peculiar change in the >> way virtual memory is reserved by the runtime : with comparable heap size >> (about 150GB) and virtual memory size (growing to 400-500GB probably due to >> a kind of fragmentation), the number of distinct memory mappings has >> apparently increased between Go 1.11 and Go 1.12 reaching the system limit >> (Linux setting vm.max_map_count). >> >> Is it something that has been experienced by someone else ? I don't believe >> this classifies as a bug, but I was a bit surprised (especially as I wasn't >> aware of that system limit). >> >> Rémy >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
