Hi, thanks to everyone for the suggestions, It seems that there is in fact not a memory leak (still not totally sure), but for some reason, the go runtime keeps allocating memory event when is not needed, I think the problem is in the interaction from the memory model of the go runtime with a Linux system with low ram (128MB).
If I force a low memory situation (by allocating memory in another program), the go runtime give back most of his memory (goes from 23 MB to 6MB), but before that happen the system become very unstable at the point that I'm unable to spawn new process (related to issue #31936), and other services start to crash(not yet sure why) or the OOMkiller start killing process. go 1.16, should "fix" this, with how it releases memory back to the system (not yet tested). I'm still a bit perplexed by the numbers from runtime. memstat, I don't help that my code does a lot of small, short-lived allocations, which makes analysis a lot harder, I will now enable memstat metric even in production hoping that will help, and working on reducing allocation in some hot path, to make the output less busy. @Tom Mitchell: yes objects plural, sorry. i will try your suggestions. a couple of notes: - I don't use CGO - I, unfortunately, can't change the environment variable, so I can use "GODEBUG=gctrace=1" only on my machine which runs/behave differently in virtue of not having the same hardware, and does not present the same memory problems/behavior, I also can't launch myself with the new variable(for a bunch of reasons that are a bit hard to explain, without going in many details), I have a couple of idea on how to fix it but I wasn't able to, - On this device run other services, that for me are black boxes, some of which seem to crash in low memory situation but I don't know yet why (i emailed the respective developers and awaiting a response) I will update, if I find something. Again thanks to everyone. On Thursday, February 11, 2021 at 6:27:00 PM UTC+1 ren...@ix.netcom.com wrote: > Are you using CGo or libraries that use CGo? If so, you probably have an > off-heap, i.e malloc() leak > > On Feb 11, 2021, at 11:11 AM, Tom Mitchell <mi...@niftyegg.com> wrote: > > > > > On Thu, Feb 11, 2021 at 3:40 AM Uli Kunitz <uli.k...@gmail.com> wrote: > >> You are writing: The device crashes with out of memory. What does crash? >> The Go program, another program or the kernel? Can you share the error >> message? Can you share the log file for one day of GODEBUG=gctrace=1? >> >> In bash do: >> >> $ export GODEBUG=gctrace=1 >> $ <your-program-that-calls-the-Go-program> 2>error.log >> >> On Thursday, February 11, 2021 at 9:01:52 AM UTC+1 Miles Hex wrote: >> >>> To be honest, i'm not totally sure, what is see is that memory keep >>> raising until the device crash for out of memory, but what does it mean if >>> reachable object remain constant, but the " Heap objects" keep raising? >>> >>>> <https://dave.cheney.net/tag/godebug> >>>> >>> > Objects plural? > What is the size of each heap object. With garbage collection it can help > to have as few sizes as possible. Pad objects so when collected the free > leaves a useful size space in the heap. Linux can allow the VM system to > over allocate resources and only discover when written and the page fault > has no backing store. Add swap to file and or tune the VM system. Limits > -- set limits low enough for the program so it cannot dominate system > physical resources. > Add signal handler code to dump a summary of all your live objects. > Recursion? Socket(state)? Events arrive faster than event handling? > Locks/Mutex: is housekeeping getting CPU time [phone booth rule, give > priority to leaving]? > > > -- > T o m M i t c h e l l ( o n N i f t y E g g ) > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAAMy4UQWV1Gt9UZBpjpAx01mhL5HEr8NCtGX1rnr1Hevui1Eiw%40mail.gmail.com > > <https://groups.google.com/d/msgid/golang-nuts/CAAMy4UQWV1Gt9UZBpjpAx01mhL5HEr8NCtGX1rnr1Hevui1Eiw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0011fc54-869f-4967-9791-2508c382a992n%40googlegroups.com.
