Because they are tracked separately. See allocSpan() in mheap.go. > On Mar 27, 2022, at 11:50 AM, Shlomi Amit <shlo...@gmail.com> wrote: > > Thank Robert. > My C code does call back Go code, but no recursion there. > If stack allocations are part of the heap, why it is not reflected in > heapAlloc and pprof? > > On Sun, Mar 27, 2022, 18:52 robert engels <reng...@ix.netcom.com > <mailto:reng...@ix.netcom.com>> wrote: > Neither of those track C allocations - unless the C is calling back into Go. > > You can review the low level usage in stack.go > > Note that Go stack allocations are for the most part done in the heap, and > are dynamically managed across Go routines. > > Are you possibly doing some very deep recursions are times? Because the > stacks will not shrink under certain conditions. > > Since they are allocated on the heap, you may be encountering a situation > where you are extended the stack to a degree that the GC cannot keep up > >> On Mar 26, 2022, at 4:01 PM, Shlomi Amit <shlo...@gmail.com >> <mailto:shlo...@gmail.com>> wrote: >> >> I've already checked process threads using ps command, and threads are not >> leaking. >> Still, it is more than possible that there are memory leaks within my Cgo >> code, or gstreamer as well (Which I'm running with Cgo as well) >> >> But before I'm starting to chase this path, I would like to be sure heapSys >> or StackInUse stats are also including Cgo memory allocations, because my >> understanding was that Go runtime is not reflecting memory allocations done >> from C code. >> >> I've aleady found Go memory leak before with pprof (Due to leak seen in >> heapObjects count) and another leak in goRoutine and fix them... But now I'm >> with the mystery of pprof showing low memory usage but stackInUse & HeapSys >> are increasing and are not reflected in pprof. >> >> So - Are HeapSys or Stack In use includes memory allocations done in C code? >> If so, I'll move on and start chasing C code (Possibly using Valgrind? Can I >> use it to profile my Go application with Cgo?) >> >> On Sat, Mar 26, 2022, 22:54 robert engels <reng...@ix.netcom.com >> <mailto:reng...@ix.netcom.com>> wrote: >> Are you certain your CGo code isn’t creating threads? >> >>> On Mar 26, 2022, at 1:10 PM, Shlomi Amit <shlo...@gmail.com >>> <mailto:shlo...@gmail.com>> wrote: >>> >>> Yes. I already monitoring the runtime stat of number of go routines (Can be >>> seen in the screenshot as well) and it's not increasing. >>> >>> On Sat, Mar 26, 2022, 20:40 robert engels <reng...@ix.netcom.com >>> <mailto:reng...@ix.netcom.com>> wrote: >>> Are you certain the number of Go routines is not increasing - each Go >>> routine requires stack. See https://tpaschalis.github.io/goroutines-size/ >>> <https://tpaschalis.github.io/goroutines-size/> >>> >>>> On Mar 24, 2022, at 3:18 PM, Shlomi Amit <shlo...@gmail.com >>>> <mailto:shlo...@gmail.com>> wrote: >>>> >>>> Hi. >>>> >>>> I’m trying to find a memory leak in my application. >>>> I’ve added some runtime memory stats to the logs (Heap & Stack stats.. go >>>> routine and heapObject counts). >>>> I can see that from time to time m.StackInuse is increasing without any >>>> obvious reason from the application perspective (At least not one that I >>>> can find yet). >>>> Here some additional runtime mem stats (Taken at the same time): >>>> StackInUse 56MB >>>> HeapSys: 147MB. >>>> HeapAlloc: 97MB >>>> >>>> While StackInUse and HeapSys seems to increase from time to time, >>>> HeapAlloc stays about the same during 6 hours while the application is >>>> running. >>>> There seems to be a bit of correlation between StackInUse and HeapSys, but >>>> overtime StackInUse increase more than HeapSys. (Is HeapSize includes >>>> StackInUse?) >>>> >>>> In addition to adding runtime memory logs, I’m also creating periodic heap >>>> profile dumps. >>>> My problem is that when analyzing the heap with pprof, it gives me no clue >>>> why StackInUse is so high. >>>> The pprof inuse_space shows: >>>> “Showing nodes accounting for 55.31MB, 93.25% of 59.31MB total” >>>> What this total MB represent? It doesn’t seem to match HeapAlloc, HeapSys >>>> or StackInUse. >>>> Does pprof heap profile even include StackInUse? >>>> >>>> I really need to understand where the leak is coming from, but after >>>> looking in many places, the memory stats are still not clear to me, and >>>> neither what memory stat pprof heap profile really represent. >>>> Note that I’m also logging HeapObjects count and I don’t see any leak >>>> there… It’s just StackInUse increasing from time to time (As it seems, >>>> it's always double itself... ~8MB->16MB->32), and HeapSys. >>>> >>>> Note that I’m also using Cgo, but my understanding is Cgo memory >>>> allocations will not be reflected by the runtime memory stats. Is this >>>> correct and I assume if runtime memory stats are increasing this is >>>> defiantly because of go code and not C code? >>>> I hope I was clear, but added a screenshot for the different memory stats. >>>> Marked in red the point in times that stackInUse increase. (My current >>>> understanding which might be wrong, is that stackInUse is not included in >>>> HeapSys, this is why they are stacked in the graph). >>>> >>>> I know my write is a bit messed up and you might not really be sure what's >>>> being asked, so in if I'll try to summarize: >>>> >>>> >>>> What stackInUse represents? Is it part of HeapSys? >>>> What HeapSys represents? (Both it and StackInUse are way more high than >>>> heapAlloc) >>>> Why pprof inuse_space doesn't seem to have any notion of stuff which was >>>> allocated by StackInUse or HeapSys? It seems to pretty much corelate with >>>> HeapAlloc. >>>> Since I'm also using Cgo, I do understand C memory allocation will not be >>>> reflected in pprof, perhaps they are being reflected by StackInUse or >>>> HeapInUse? >>>> Considering StackInUse seems to double itself in size when it increase, it >>>> seems like some kind of memory pre-allocation to have enough headroom than >>>> needed, not something which I expect to happen by C code. >>>> Most importantly, with the below memory stats being increased, what's the >>>> approach to investigate? >>>> Currently I'm using docker, and sometimes get OOM. What's best memory >>>> stats which will the most close to describe container RSS memory in use? >>>> (Specifically, my docker container was limited to 300MiB) >>>> >>>> I know these are many question, but I did try and go over the runtime >>>> memory stats documentation, and I'm still very confused, especially not >>>> after the actual memory stats I'm seeing. >>>> BTW, I'm using go 1.17.5 >>>> >>>> <Untitled.jpg> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to golang-nuts+unsubscr...@googlegroups.com >>>> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> <Untitled.jpg> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to golang-nuts+unsubscr...@googlegroups.com >>> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com >> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/CAAbwE0O6Wj2ngcGymiJhrwUxOJLQ_5TZbJgqWnqm_c_BZc8-bQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/golang-nuts/CAAbwE0O6Wj2ngcGymiJhrwUxOJLQ_5TZbJgqWnqm_c_BZc8-bQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. >
-- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/B2B93478-D0FE-4325-9444-AA3FA13A0635%40ix.netcom.com.
