Hello, I didn't have time to try the patch, but as Go 1.12.5 is out with another fix, I can confirm with high certainty that the problem is now gone, and I am pretty much confident that the situation is even better than Go 1.11, which is consistent with the fact that the bug was already there, but milder because it only applied to large allocations.
Thanks a lot, Rémy Le mar. 16 avr. 2019 à 19:11, Michael Knyszek <mknys...@google.com> a écrit : > Hey Rémy, > > If you have a chance, could you please try out this patch > <https://go-review.googlesource.com/c/go/+/151538/23>? It's been known to > help the other application Austin mentioned with virtual memory footprint > and it should patch cleanly onto the go1.12. Let me know what you see! It'd > help us to confirm the root cause of the VSS growth. > > Thanks, > Michael > > On Tue, Apr 16, 2019 at 11:03 AM Austin Clements <aus...@google.com> > wrote: > >> On Tue, Apr 16, 2019 at 1:23 AM Rémy Oudompheng <remyoudomph...@gmail.com> >> wrote: >> >>> Thanks Austin, >>> >>> The application workload is a kind of fragmentation torture test as it >>> involves a mixture of many long-lived small and large (>100 MB) >>> objects, with regularly allocated short-lived small and large objects. >>> I have tried creating a sample synthetic reproducer but did not >>> succeed at the moment. >>> >>> Regarding the max_map_count, your explanation is very clear and I >>> apparently missed the large comment in the runtime explaining all of >>> that. >>> Do you expect a significant drawback between choosing 2MB or 16MB as >>> the granularity of the huge page flag manipulation in the case of huge >>> heaps ? >>> >> >> Most likely this will just cause less use of huge pages in your >> application. This could slow it down by putting more pressure on the TLB. >> In a sense, this is a self-compounding issue since huge pages can be highly >> beneficial to huge heaps. >> >> Regarding the virtual memory footprint, it changed radically with Go >>> 1.12. It basically looks like a leak and I saw it grow to more than >>> 1TB where the actual heap total size never exceeds 180GB. >>> Although I understand that it is easy to construct a situation where >>> there is repeatedly no available contiguous interval of >100MB in the >>> address space, it is a significant difference from Go 1.11 where the >>> address space would grow to 400-500GB for a similar workload and stay >>> flat after that, and I could not find an obvious change in the >>> allocator explaining the phenomenon (and unfortunately my resources do >>> not allow for an easy live comparison of both program lifetimes). >>> >>> Am I right saying that scavenging method or frequency does not >>> (cannot) affect at all virtual memory footprint and dynamics ? >>> >> >> It certainly can affect virtual memory footprint because of how >> scavenging affects the allocator's placement policy. Though even with the >> increased VSS, I would expect your application to have lower RSS with 1.12. >> In almost all cases, lower RSS with higher VSS is a fine trade-off, though >> lower RSS with the same VSS would obviously be better. But it can be >> problematic when it causes the map count (which is roughly proportional to >> the VSS) to grow too large. It's also unfortunate that Linux even has this >> limit; it's the only OS Go runs on that limits the map count. >> >> We've seen one other application experience VSS growth with the 1.12 >> changes, and it does seem to require a pretty unique allocation pattern. >> Michael (cc'd) may be zeroing in on the causes of this and may have some >> patches for you to try if you don't mind. :) >> >> Regards, >>> Rémy. >>> >>> Le mar. 2 avr. 2019 à 16:15, Austin Clements <aus...@google.com> a >>> écrit : >>> > >>> > Hi Rémy. We often fight with vm.max_map_count in the runtime, sadly. >>> Most likely this comes from the way the runtime interacts with Linux's >>> transparent huge page support. When we scavenge (release to the OS) only >>> part of a huge page, we tell the OS not to turn that huge page frame back >>> into a huge page since that would just make that memory used again. >>> Unfortunately, each time we do this counts as a separate "mapping" just to >>> track that one flag. These "mappings" are always at least 2MB, but you have >>> a large enough virtual address space to reach the max_map_count even then. >>> You can see this in /proc/PID/smaps, which should list mostly contiguous >>> neighboring regions that differ only in a single "VmFlags" bit. >>> > >>> > We did make memory scavenging more aggressive in Go 1.12 (+Michael >>> Knyszek), though I would have expected it to converge to roughly the same >>> "huge page flag fragmentation" as before over the course of five to ten >>> minutes. Is your application's virtual memory footprint the same between >>> 1.11 and 1.12 or do that grow? >>> > >>> > You could try disabling the huge page flag manipulation to confirm >>> and/or fix this. In $GOROOT/src/runtime/internal/sys/arch_amd64.go (or >>> whichever GOARCH is appropriate), set HugePageSize to 0. Though there's a >>> danger that Linux's transparent huge pages could blow up your application's >>> resident set size if you do that. >>> > >>> > On Tue, Apr 2, 2019 at 3:49 AM Rémy Oudompheng < >>> remyoudomph...@gmail.com> wrote: >>> >> >>> >> Hello, >>> >> >>> >> In a large heap program I am working on, I noticed a peculiar change >>> in the way virtual memory is reserved by the runtime : with comparable heap >>> size (about 150GB) and virtual memory size (growing to 400-500GB probably >>> due to a kind of fragmentation), the number of distinct memory mappings has >>> apparently increased between Go 1.11 and Go 1.12 reaching the system limit >>> (Linux setting vm.max_map_count). >>> >> >>> >> Is it something that has been experienced by someone else ? I don't >>> believe this classifies as a bug, but I was a bit surprised (especially as >>> I wasn't aware of that system limit). >>> >> >>> >> Rémy >>> >> >>> >> -- >>> >> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> >> To unsubscribe from this group and stop receiving emails from it, >>> send an email to golang-nuts+unsubscr...@googlegroups.com. >>> >> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CADEq3_ud9ADM%2BVe_Q%3DrVKDfEO%3DOEfRCP9ziWkjzQu2_%2BTRtPig%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
