I suspect your implementation is incorrect in some way, or you have a memory leak of another sort.
When I used your original code, the memory usage according to the activity monitor was > 100 mb. When I changed it to use a semaphore to limit the concurrency to 64 threads, it was about 16mb. You also have to be aware that Go does not have a compacting collector, so depending on your allocation pattern it may be nearly impossible to release the physical pages back to the OS. It can release physical pages back to the OS if the garbage collector can determine that nothing in the address range is being used. So bottom line, you have to be careful about which memory statistics you use to “measure memory” with Go. This has a lot of great information here https://tip.golang.org/doc/gc-guide#A_note_about_virtual_memory > On Mar 8, 2025, at 12:35 PM, David Bell <davidbellou...@gmail.com> wrote: > > I wanted to add some clarification on my misunderstanding and the approaches > I tried. > > I tested two different strategies to handle the CGO calls: > > Worker Pool Approach (50 workers handling CGO calls) > > I set up a worker pool with 50 goroutines, each making CGO calls in a > controlled manner. > Surprisingly, this did not resolve the issue—memory usage still kept > increasing. > Locking OS Threads with a Semaphore (Up to 500 concurrent CGO calls) > > Instead of using a worker pool, I tried having each CGO call run in its own > goroutine but with runtime.LockOSThread(). > I also used a semaphore to ensure that at most 500 CGO calls could run > concurrently. > This approach did resolve the issue—memory usage stopped growing indefinitely. > What I don’t quite understand is why the second approach works while the > first one doesn’t. > If the worker pool limits concurrency to 50 workers, why does it still cause > memory growth, while 500 locked threads behave better? Is it because of how > Go manages OS threads internally? > > Appreciate your thoughts on this! > > On Saturday, 8 March 2025 at 20:24:12 UTC+2 David Bell wrote: >> Hi Ian and Robert, >> >> First, thank you both for taking the time to answer and provide insights! I >> really appreciate it. >> >> To clarify, my real program is a long-lived process that receives network >> traffic from C++ code and processes it in Go. >> The CGO calls in my program seemed to be the likely culprit behind the >> increasing memory usage I observed. >> >> I tried adding runtime.LockOSThread() around the CGO calls and used a >> semaphore to limit concurrency. >> This change has significantly improved the issue, as the memory growth is >> now much more controlled. >> >> However, there’s something I still don’t fully understand. In my toy >> example, where I spawn 5000 CGO calls, locking the thread indeed causes Go >> to properly clean up the threads, and I end up with the correct number of >> threads. But despite that, memory usage still remains around 200MB. >> >> If locking the thread ensures proper cleanup, why does memory usage remain >> high in the toy example, while in my real program, it seems to help? Is >> there some underlying mechanism in Go’s thread management or memory >> allocator that could explain this difference? >> >> Thanks again for your help! >> >> >> On Tuesday, 4 March 2025 at 23:16:37 UTC+2 robert engels wrote: >>> How many threads are you limiting it to? The OS thread stacks can be large. >>> A basic premise of Go is to not use threads, and use Go routines - harder >>> with CGO - but you often need to think about rearchitecting to make the C >>> layer a “service handler” - like any other service - and then limit the >>> number of requests in flight. >>> >>> When I limit it to 64 threads on my Mac, I see about 16mb memory being used. >>> >>> >>>> On Mar 4, 2025, at 2:59 PM, David Bell <davidbe...@gmail.com <>> wrote: >>>> >>> >>>> Thanks for the response Robert, >>>> >>>> I tried to put a semaphore to control the max amount CGO calls >>>> simultaneously, >>>> similar to what they did here >>>> https://github.com/golang/go/blob/master/src/net/net.go#L794-L818 >>>> >>>> i noticed it does help somewhat, but memory consumption keeps going up >>>> uncontrollably. >>>> >>>> >>>> >>>> I tried to do LockOSThread and use GoExit so the thread will close. >>>> it indeed closes the thread but the memory is never released back to the >>>> os. >>>> >>>> is there a way i can force it to release the resources back to the os when >>>> the thread is closed? >>>> ב-יום שלישי, 4 במרץ 2025 בשעה 22:32:45 UTC+2, robert engels כתב/ה: >>>>> Or at the github issue points out, if the thread will exit, LockOSThread >>>>> my work. >>>>> >>>>> On Tuesday, March 4, 2025 at 2:23:28 PM UTC-6 robert engels wrote: >>>>>> One way you can address this is to put a semaphore on the Go side around >>>>>> the C call, so you ensure only so many C calls are made simultaneously. >>>>>> >>>>>> >>>>>> >>>>>>> On Mar 4, 2025, at 1:26 PM, David Bell <davidbe...@gmail.com <>> wrote: >>>>>>> >>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> I'm relatively new to Go and even newer to CGO, so I’d really >>>>>>> appreciate any guidance on an issue I’ve been facing. >>>>>>> >>>>>>> Problem Overview >>>>>>> >>>>>>> I noticed that when using CGO, my application's memory usage keeps >>>>>>> increasing, and threads do not seem to be properly cleaned up. >>>>>>> The Go runtime (pprof) does not indicate any leaks, but when I monitor >>>>>>> the process using Activity Monitor, I see a growing number of threads >>>>>>> and increasing memory consumption. >>>>>>> >>>>>>> Observations >>>>>>> Memory usage keeps increasing over time when using CGO, even after >>>>>>> forcing garbage collection. >>>>>>> Threads spawned for CGO calls appear to not be released, causing a >>>>>>> large number of lingering threads. >>>>>>> The issue is not present when using pure Go time.Sleep() instead of the >>>>>>> CGO function. >>>>>>> Reproducible Example >>>>>>> >>>>>>> I created a minimal program that reproduces the issue. The following >>>>>>> CGO-based code keeps allocating memory and does not free threads >>>>>>> properly: >>>>>>> >>>>>>> package main >>>>>>> >>>>>>> import ( >>>>>>> "fmt" >>>>>>> "runtime" >>>>>>> "runtime/debug" >>>>>>> "sync" >>>>>>> "time" >>>>>>> ) >>>>>>> >>>>>>> /* >>>>>>> #include <unistd.h> >>>>>>> void cgoSleep() { >>>>>>> sleep(1); >>>>>>> } >>>>>>> */ >>>>>>> import "C" >>>>>>> >>>>>>> func main() { >>>>>>> start := time.Now() >>>>>>> >>>>>>> var wg sync.WaitGroup >>>>>>> for i := 0; i < 5000; i++ { >>>>>>> wg.Add(1) >>>>>>> go func() { >>>>>>> defer wg.Done() >>>>>>> C.cgoSleep() >>>>>>> }() >>>>>>> } >>>>>>> wg.Wait() >>>>>>> >>>>>>> end := time.Now() >>>>>>> >>>>>>> // Force GC and free OS memory >>>>>>> runtime.GC() >>>>>>> debug.FreeOSMemory() >>>>>>> time.Sleep(10 * time.Second) >>>>>>> >>>>>>> var m runtime.MemStats >>>>>>> runtime.ReadMemStats(&m) >>>>>>> >>>>>>> fmt.Printf("Alloc = %v MiB", m.Alloc/1024/1024) >>>>>>> fmt.Printf("\tTotalAlloc = %v MiB", m.TotalAlloc/1024/1024) >>>>>>> fmt.Printf("\tSys = %v MiB", m.Sys/1024/1024) >>>>>>> fmt.Printf("\tNumGC = %v\n", m.NumGC) >>>>>>> fmt.Printf("Total time: %v\n", end.Sub(start)) >>>>>>> >>>>>>> select {} >>>>>>> } >>>>>>> >>>>>>> Expected Behavior >>>>>>> The memory usage should not continue rising indefinitely. >>>>>>> Threads should be properly cleaned up when they finish executing. >>>>>>> The behavior should be similar to the following pure Go equivalent, >>>>>>> which does not exhibit the issue: >>>>>>> >>>>>>> >>>>>>> Actual Results With CGO (cgoSleep()): >>>>>>> Memory Usage: 296 MB >>>>>>> Threads: 5,003 >>>>>>> System Memory (Sys from runtime.MemStats): 205 MB >>>>>>> >>>>>>> >>>>>>> With Pure Go (time.Sleep()): >>>>>>> Memory Usage: 14 MB >>>>>>> Threads: 14 >>>>>>> System Memory (Sys from runtime.MemStats): 24 MB >>>>>>> Additional Attempt >>>>>>> I tried forcing thread cleanup using runtime.LockOSThread() and >>>>>>> runtime.Goexit(), but while the number of threads decreases, memory is >>>>>>> still never fully released: >>>>>>> >>>>>>> go func() { runtime.LockOSThread() defer wg.Done() C.cgoSleep() >>>>>>> runtime.Goexit() }() Questions >>>>>>> Why is memory increasing indefinitely with CGO? >>>>>>> Why are threads not getting properly cleaned up after CGO calls? >>>>>>> Is there a way to force the Go runtime to reclaim memory allocated for >>>>>>> CGO threads? >>>>>>> Is there a better approach to handling CGO calls that spawn short-lived >>>>>>> threads? >>>>>>> Would using runtime.UnlockOSThread() help in this case, or is this >>>>>>> purely a CGO threading issue? >>>>>>> Is there a way to track down where the memory is being held? Since >>>>>>> pprof does not show high memory usage, what other tools can I use? >>>>>>> Go Version >>>>>>> go1.23.5 darwin/arm64 >>>>>>> >>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "golang-nuts" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to golang-nuts...@googlegroups.com <>. >>>>>>> To view this discussion visit >>>>>>> https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to golang-nuts...@googlegroups.com <>. >>> >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/golang-nuts/9a40383e-5b47-4f2b-bdea-a50946381650n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/golang-nuts/9a40383e-5b47-4f2b-bdea-a50946381650n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> > > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com > <mailto:golang-nuts+unsubscr...@googlegroups.com>. > To view this discussion visit > https://groups.google.com/d/msgid/golang-nuts/82290274-54d4-41c5-8c9f-4d881e6b474dn%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/82290274-54d4-41c5-8c9f-4d881e6b474dn%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/48070E6D-DDE1-4F9C-A807-58C11C4CF255%40ix.netcom.com.
