Hi all, Under what circumstances could I expect the Go GC time per pause (right now calculated using PauseTotalNS / NumGC using the runtime stats) creep up slowly over time?
Assuming the number of allocations we do per second is consistent, could it be that the GC is somehow not keeping up with our allocations? I'm not familiar enough with the GC internals (yet) to know if this is even possible. For example, if we allocate 10k things, is it possible that only 8k of those things are collected, and then next cycle it'll have 12k things to clean up? (numbers used as examples only) I don't believe there is a memory leak because our memory utilization (as reported by Go runtime and the system) is steady after a warmup period. Our situation: - go version go1.7.4 linux/amd64 - HTTP Server with timeouts configured serving JSON post requests and JSON responses on most endpoints (2 endpoints serving protobuf requests/responses) - 30-40k QPS per node with an approximately consistent number of allocations per request. - We are seeing the GC time per pause creep up slowly over over 24 hours. - The rate at which the GC time increases over time is like clockwork (3.5ms per pause to 6.5ms per pause). - The rate continues to increase after that, but we restart the process daily which immediately resolves the problem. - HTTP Response times, as measured from the beginning of ServeHTTP to the end, increase directly with the increase in GC pause time (We go from 2.5ms mean response time to 8ms mean response time over 24 hours). - We have several nodes that handle only 15-20k QPS that either don't experience this problem, or the creep is so slow as to be not noticeable (as in we deploy frequently enough that the issue never becomes apparent). We're doing what we can to reduce allocations, which has had a significant impact (we used to go from 6ms mean to 15ms mean in 24 hours); however, no matter what we do we can't end the creep. We've run servers for several days and the response time and GC time per pause continues to go up until the response times are unacceptable and we must restart the process. Thank you! Evan -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
