Hi Ian and Robert,
First, thank you both for taking the time to answer and provide insights! I really appreciate it. To clarify, my real program is a long-lived process that receives network traffic from C++ code and processes it in Go. The CGO calls in my program seemed to be the likely culprit behind the increasing memory usage I observed. I tried adding runtime.LockOSThread() around the CGO calls and used a semaphore to limit concurrency. This change has significantly improved the issue, as the memory growth is now much more controlled. However, there’s something I still don’t fully understand. In my toy example, where I spawn 5000 CGO calls, locking the thread indeed causes Go to properly clean up the threads, and I end up with the correct number of threads. But despite that, memory usage still remains around 200MB. If locking the thread ensures proper cleanup, why does memory usage remain high in the toy example, while in my real program, it seems to help? Is there some underlying mechanism in Go’s thread management or memory allocator that could explain this difference? Thanks again for your help! On Tuesday, 4 March 2025 at 23:16:37 UTC+2 robert engels wrote: > How many threads are you limiting it to? The OS thread stacks can be > large. A basic premise of Go is to not use threads, and use Go routines - > harder with CGO - but you often need to think about rearchitecting to make > the C layer a “service handler” - like any other service - and then limit > the number of requests in flight. > > When I limit it to 64 threads on my Mac, I see about 16mb memory being > used. > > On Mar 4, 2025, at 2:59 PM, David Bell <davidbe...@gmail.com> wrote: > > Thanks for the response Robert, > > I tried to put a semaphore to control the max amount CGO calls > simultaneously, > similar to what they did here > https://github.com/golang/go/blob/master/src/net/net.go#L794-L818 > > i noticed it does help somewhat, but memory consumption keeps going up > uncontrollably. > > > > I tried to do LockOSThread and use GoExit so the thread will close. > it indeed closes the thread but the memory is never released back to the > os. > > is there a way i can force it to release the resources back to the os when > the thread is closed? > ב-יום שלישי, 4 במרץ 2025 בשעה 22:32:45 UTC+2, robert engels כתב/ה: > >> Or at the github issue points out, if the thread will exit, LockOSThread >> my work. >> >> On Tuesday, March 4, 2025 at 2:23:28 PM UTC-6 robert engels wrote: >> >>> One way you can address this is to put a semaphore on the Go side around >>> the C call, so you ensure only so many C calls are made simultaneously. >>> >>> >>> On Mar 4, 2025, at 1:26 PM, David Bell <davidbe...@gmail.com> wrote: >>> >>> *Hi everyone,* >>> >>> I'm relatively new to Go and even newer to *CGO*, so I’d really >>> appreciate any guidance on an issue I’ve been facing. >>> >>> *Problem Overview* >>> >>> I noticed that when using CGO, my application's *memory usage keeps >>> increasing*, and threads do not seem to be properly cleaned up. >>> The Go runtime (pprof) does not indicate any leaks, but when I monitor >>> the process using *Activity Monitor*, I see a growing number of threads >>> and increasing memory consumption. >>> *Observations* >>> >>> - Memory usage *keeps increasing* over time when using CGO, even >>> after forcing garbage collection. >>> - Threads spawned for CGO calls appear to *not be released*, causing >>> a large number of lingering threads. >>> - The issue is *not present* when using pure Go time.Sleep() instead >>> of the CGO function. >>> >>> >>> >>> *Reproducible Example* >>> >>> I created a minimal program that reproduces the issue. The following >>> *CGO-based* code keeps allocating memory and does not free threads >>> properly: >>> package main >>> >>> import ( >>> "fmt" >>> "runtime" >>> "runtime/debug" >>> "sync" >>> "time" >>> ) >>> >>> /* >>> #include <unistd.h> >>> void cgoSleep() { >>> sleep(1); >>> } >>> */ >>> import "C" >>> >>> func main() { >>> start := time.Now() >>> >>> var wg sync.WaitGroup >>> for i := 0; i < 5000; i++ { >>> wg.Add(1) >>> go func() { >>> defer wg.Done() >>> C.cgoSleep() >>> }() >>> } >>> wg.Wait() >>> >>> end := time.Now() >>> >>> // Force GC and free OS memory >>> runtime.GC() >>> debug.FreeOSMemory() >>> time.Sleep(10 * time.Second) >>> >>> var m runtime.MemStats >>> runtime.ReadMemStats(&m) >>> >>> fmt.Printf("Alloc = %v MiB", m.Alloc/1024/1024) >>> fmt.Printf("\tTotalAlloc = %v MiB", m.TotalAlloc/1024/1024) >>> fmt.Printf("\tSys = %v MiB", m.Sys/1024/1024) >>> fmt.Printf("\tNumGC = %v\n", m.NumGC) >>> fmt.Printf("Total time: %v\n", end.Sub(start)) >>> >>> select {} >>> } >>> >>> *Expected Behavior* >>> >>> - The memory usage should *not* continue rising indefinitely. >>> - Threads should be properly cleaned up when they finish executing. >>> - The behavior should be similar to the following *pure Go* >>> equivalent, which does *not* exhibit the issue: >>> >>> >>> >>> *Actual Results* *With CGO (cgoSleep()):* >>> >>> - *Memory Usage:* *296 MB* >>> - *Threads:* *5,003* >>> - *System Memory (Sys from runtime.MemStats)*: >>> >>> >>> *205 MB * >>> >>> *With Pure Go (time.Sleep()):* >>> >>> - *Memory Usage:* *14 MB* >>> - *Threads:* *14* >>> - *System Memory (Sys from runtime.MemStats)*: *24 MB* >>> >>> *Additional Attempt* >>> >>> I tried forcing thread cleanup using runtime.LockOSThread() and >>> runtime.Goexit(), but *while the number of threads decreases, memory is >>> still never fully released*: >>> go func() { runtime.LockOSThread() defer wg.Done() C.cgoSleep() >>> runtime.Goexit() }() *Questions* >>> >>> 1. *Why is memory increasing indefinitely with CGO?* >>> 2. *Why are threads not getting properly cleaned up after CGO calls?* >>> 3. *Is there a way to force the Go runtime to reclaim memory >>> allocated for CGO threads?* >>> 4. *Is there a better approach to handling CGO calls that spawn >>> short-lived threads?* >>> 5. *Would using runtime.UnlockOSThread() help in this case, or is >>> this purely a CGO threading issue?* >>> 6. *Is there a way to track down where the memory is being held? >>> Since pprof does not show high memory usage, what other tools can I >>> use?* >>> >>> *Go Version* >>> go1.23.5 darwin/arm64 >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > > To view this discussion visit > https://groups.google.com/d/msgid/golang-nuts/9a40383e-5b47-4f2b-bdea-a50946381650n%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/9a40383e-5b47-4f2b-bdea-a50946381650n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/09b554f4-0175-4d54-9bd0-c2420d5b996fn%40googlegroups.com.
