On Wed, Nov 16, 2022 at 10:19 AM Jie Zhang <tx.zhi...@gmail.com> wrote:
> Actually, we used go1.16.5 before go1.19 released. Now we're consider > using go1.19 GOMEMLIMIT instead. > > But GOMEMLIMIT is a soft limit, if you deploy multiple services in the > same host (not per container per service), there will be problems. > The GC is delayed, and if combined with GOGC=off, the memory usage will be > higher. > In that scenario, I might suggest setting GOGC higher than 100 (not off) but with some memory limit on each process instead. > > Maybe manually trigger forced GC, and combine GOMEMLIMIT will be better. > I'm testing... > Manually triggering GCs has a relatively high risk of hurting performance. Even if it doesn't hurt performance today, it might in the next release, unless used very carefully. I don't recommend it outside of testing. > > 在2022年11月11日星期五 UTC+8 05:58:54<mkny...@google.com> 写道： > >> That's correct. The runtime has a simple heuristic for avoiding zeroing >> but it's far from perfect. As a result, a ballast is inherently always >> going to be a little risky. This is especially true on some platforms, such >> as Windows, since there's no way to avoid marking the memory as committed >> (Windows is free to use demand paging for memory in the range, so overall >> system memory pressure may increase, but you can't avoid it being >> *counted* as committed for a particular process). >> >> Taking a step back: why a ballast? What about your application makes a >> ballast a better idea than, for example, setting GOMEMLIMIT=<something> and >> GOGC=off? >> >> (For additional context, back when the memory limit was proposed, so was >> a memory target ( >> https://go.googlesource.com/proposal/+/master/design/44309-user-configurable-memory-target.md) >> which more directly replaces a ballast. I found very little interest in >> this feature.) >> >> On Monday, November 7, 2022 at 9:08:23 AM UTC-5 hit.zh...@gmail.com >> wrote: >> >>> Hi, guys, I know what happened. >>> >>> When we write `ballast := make([]byte, 1<<30)`, it will call makeslice >>> to create a new slice. It's a large object. The memory will be allocated >>> directly via allocLarge() function from heap. >>> >>> Actually, after allocating a mspan for it, it will check the address >>> range whether it should be zeroed. Please check function `func >>> allocNeedsZero(base, npage uintptr) bool`. When this function returns true, >>> it means the slice underlying memory will be written to zero, otherwise it >>> won't write it to zero. >>> >>> When we make a ballast as mentioned before, maybe it succeed (no RSS >>> taken up) or fail (RSS taken up), it's relevant with the return value of >>> function allocNeedsZero(...). And this function return true or false is >>> relevant with the arena's state which is affected by previous object >>> allocation and recycle. >>> >>> On Monday, November 7, 2022 at 1:26:41 PM UTC+8 tapi...@gmail.com wrote: >>> >>>> I ever also found this problem: global ballast doesn't work. >>>> So I always use local ballast instead now. >>>> >>>> On Sunday, November 6, 2022 at 8:37:55 PM UTC+8 hit.zh...@gmail.com >>>> wrote: >>>> >>>>> Before, I think the memory is allocated by mmap anon, which Linux OS >>>>> guarantees reading will return zero value and no physical pages allocated. >>>>> When writing happens, the physical pages will be allocated. Then I think >>>>> the create a byte slice maybe the same. >>>>> >>>>> Your idea is clear. I agree with it. >>>>> >>>>> Just now, I use gdb to dump the ballast anon memory and use hexdump to >>>>> check its dirty content, all data is zero (Maybe zeroing happens). >>>>> But after turning GC off, it works as expected (no RSS is taken, no >>>>> DIRTY). >>>>> I think there must be something I didn't get it. >>>>> >>>>> On Sunday, November 6, 2022 at 8:11:51 PM UTC+8 Jan Mercl wrote: >>>>> >>>>>> On Sun, Nov 6, 2022 at 12:54 PM Kn (Kn) <hit.zh...@gmail.com> wrote: >>>>>> >>>>>> > Now the problem begins. I expect the ballast like `ballast := >>>>>> make([]byte, 1<<30)` shouldn't take up any physical memory because >>>>>> there's >>>>>> no any writing to it. >>>>>> >>>>>> The backing array is specified to be zeroed, so we cannot say there's >>>>>> no writing to it. Depending on the size of the backing array and the >>>>>> operating system it may not get written as an optimization if backed >>>>>> by memory the OS can guarantee to be zero filled. Only then it may >>>>>> remain not yet bound to physical memory. >>>>>> >>>>>> A simple implementation will just zero it, meaning the opposite >>>>>> happens - every byte of the backing array gets written and backing >>>>>> pages for it get allocated. >>>>>> >>>>> -- > You received this message because you are subscribed to a topic in the > Google Groups "golang-nuts" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/golang-nuts/66d0cItfkjY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/858a4a4d-d049-40d7-9efe-ee6cfae0fa2dn%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/858a4a4d-d049-40d7-9efe-ee6cfae0fa2dn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAFza%2Bu_T7dHsuc7tn1%3DPhentuaaGdfKcEejB89NWrOXesZtnTw%40mail.gmail.com.
