Neither of those track C allocations - unless the C is calling back into Go.
You can review the low level usage in stack.go Note that Go stack allocations are for the most part done in the heap, and are dynamically managed across Go routines. Are you possibly doing some very deep recursions are times? Because the stacks will not shrink under certain conditions. Since they are allocated on the heap, you may be encountering a situation where you are extended the stack to a degree that the GC cannot keep up > On Mar 26, 2022, at 4:01 PM, Shlomi Amit <shlo...@gmail.com> wrote: > > I've already checked process threads using ps command, and threads are not > leaking. > Still, it is more than possible that there are memory leaks within my Cgo > code, or gstreamer as well (Which I'm running with Cgo as well) > > But before I'm starting to chase this path, I would like to be sure heapSys > or StackInUse stats are also including Cgo memory allocations, because my > understanding was that Go runtime is not reflecting memory allocations done > from C code. > > I've aleady found Go memory leak before with pprof (Due to leak seen in > heapObjects count) and another leak in goRoutine and fix them... But now I'm > with the mystery of pprof showing low memory usage but stackInUse & HeapSys > are increasing and are not reflected in pprof. > > So - Are HeapSys or Stack In use includes memory allocations done in C code? > If so, I'll move on and start chasing C code (Possibly using Valgrind? Can I > use it to profile my Go application with Cgo?) > > On Sat, Mar 26, 2022, 22:54 robert engels <reng...@ix.netcom.com > <mailto:reng...@ix.netcom.com>> wrote: > Are you certain your CGo code isn’t creating threads? > >> On Mar 26, 2022, at 1:10 PM, Shlomi Amit <shlo...@gmail.com >> <mailto:shlo...@gmail.com>> wrote: >> >> Yes. I already monitoring the runtime stat of number of go routines (Can be >> seen in the screenshot as well) and it's not increasing. >> >> On Sat, Mar 26, 2022, 20:40 robert engels <reng...@ix.netcom.com >> <mailto:reng...@ix.netcom.com>> wrote: >> Are you certain the number of Go routines is not increasing - each Go >> routine requires stack. See https://tpaschalis.github.io/goroutines-size/ >> <https://tpaschalis.github.io/goroutines-size/> >> >>> On Mar 24, 2022, at 3:18 PM, Shlomi Amit <shlo...@gmail.com >>> <mailto:shlo...@gmail.com>> wrote: >>> >>> Hi. >>> >>> I’m trying to find a memory leak in my application. >>> I’ve added some runtime memory stats to the logs (Heap & Stack stats.. go >>> routine and heapObject counts). >>> I can see that from time to time m.StackInuse is increasing without any >>> obvious reason from the application perspective (At least not one that I >>> can find yet). >>> Here some additional runtime mem stats (Taken at the same time): >>> StackInUse 56MB >>> HeapSys: 147MB. >>> HeapAlloc: 97MB >>> >>> While StackInUse and HeapSys seems to increase from time to time, HeapAlloc >>> stays about the same during 6 hours while the application is running. >>> There seems to be a bit of correlation between StackInUse and HeapSys, but >>> overtime StackInUse increase more than HeapSys. (Is HeapSize includes >>> StackInUse?) >>> >>> In addition to adding runtime memory logs, I’m also creating periodic heap >>> profile dumps. >>> My problem is that when analyzing the heap with pprof, it gives me no clue >>> why StackInUse is so high. >>> The pprof inuse_space shows: >>> “Showing nodes accounting for 55.31MB, 93.25% of 59.31MB total” >>> What this total MB represent? It doesn’t seem to match HeapAlloc, HeapSys >>> or StackInUse. >>> Does pprof heap profile even include StackInUse? >>> >>> I really need to understand where the leak is coming from, but after >>> looking in many places, the memory stats are still not clear to me, and >>> neither what memory stat pprof heap profile really represent. >>> Note that I’m also logging HeapObjects count and I don’t see any leak >>> there… It’s just StackInUse increasing from time to time (As it seems, it's >>> always double itself... ~8MB->16MB->32), and HeapSys. >>> >>> Note that I’m also using Cgo, but my understanding is Cgo memory >>> allocations will not be reflected by the runtime memory stats. Is this >>> correct and I assume if runtime memory stats are increasing this is >>> defiantly because of go code and not C code? >>> I hope I was clear, but added a screenshot for the different memory stats. >>> Marked in red the point in times that stackInUse increase. (My current >>> understanding which might be wrong, is that stackInUse is not included in >>> HeapSys, this is why they are stacked in the graph). >>> >>> I know my write is a bit messed up and you might not really be sure what's >>> being asked, so in if I'll try to summarize: >>> >>> >>> What stackInUse represents? Is it part of HeapSys? >>> What HeapSys represents? (Both it and StackInUse are way more high than >>> heapAlloc) >>> Why pprof inuse_space doesn't seem to have any notion of stuff which was >>> allocated by StackInUse or HeapSys? It seems to pretty much corelate with >>> HeapAlloc. >>> Since I'm also using Cgo, I do understand C memory allocation will not be >>> reflected in pprof, perhaps they are being reflected by StackInUse or >>> HeapInUse? >>> Considering StackInUse seems to double itself in size when it increase, it >>> seems like some kind of memory pre-allocation to have enough headroom than >>> needed, not something which I expect to happen by C code. >>> Most importantly, with the below memory stats being increased, what's the >>> approach to investigate? >>> Currently I'm using docker, and sometimes get OOM. What's best memory stats >>> which will the most close to describe container RSS memory in use? >>> (Specifically, my docker container was limited to 300MiB) >>> >>> I know these are many question, but I did try and go over the runtime >>> memory stats documentation, and I'm still very confused, especially not >>> after the actual memory stats I'm seeing. >>> BTW, I'm using go 1.17.5 >>> >>> <Untitled.jpg> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to golang-nuts+unsubscr...@googlegroups.com >>> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> <Untitled.jpg> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com >> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com > <mailto:golang-nuts+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAAbwE0O6Wj2ngcGymiJhrwUxOJLQ_5TZbJgqWnqm_c_BZc8-bQ%40mail.gmail.com > > <https://groups.google.com/d/msgid/golang-nuts/CAAbwE0O6Wj2ngcGymiJhrwUxOJLQ_5TZbJgqWnqm_c_BZc8-bQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1CDA55C7-9028-4133-B2B1-FD28C26CF325%40ix.netcom.com.
