Thanks for the response Robert,
I tried to put a semaphore to control the max amount CGO calls
simultaneously,
similar to what they did here
https://github.com/golang/go/blob/master/src/net/net.go#L794-L818
i noticed it does help somewhat, but memory consumption keeps going up
uncontrollably.
I tried to do LockOSThread and use GoExit so the thread will close.
it indeed closes the thread but the memory is never released back to the os.
is there a way i can force it to release the resources back to the os when
the thread is closed?
ב-יום שלישי, 4 במרץ 2025 בשעה 22:32:45 UTC+2, robert engels כתב/ה:
> Or at the github issue points out, if the thread will exit, LockOSThread
> my work.
>
> On Tuesday, March 4, 2025 at 2:23:28 PM UTC-6 robert engels wrote:
>
>> One way you can address this is to put a semaphore on the Go side around
>> the C call, so you ensure only so many C calls are made simultaneously.
>>
>>
>> On Mar 4, 2025, at 1:26 PM, David Bell <davidbe...@gmail.com> wrote:
>>
>> *Hi everyone,*
>>
>> I'm relatively new to Go and even newer to *CGO*, so I’d really
>> appreciate any guidance on an issue I’ve been facing.
>>
>> *Problem Overview*
>>
>> I noticed that when using CGO, my application's *memory usage keeps
>> increasing*, and threads do not seem to be properly cleaned up.
>> The Go runtime (pprof) does not indicate any leaks, but when I monitor
>> the process using *Activity Monitor*, I see a growing number of threads
>> and increasing memory consumption.
>> *Observations*
>>
>> - Memory usage *keeps increasing* over time when using CGO, even
>> after forcing garbage collection.
>> - Threads spawned for CGO calls appear to *not be released*, causing
>> a large number of lingering threads.
>> - The issue is *not present* when using pure Go time.Sleep() instead
>> of the CGO function.
>>
>>
>>
>> *Reproducible Example*
>>
>> I created a minimal program that reproduces the issue. The following
>> *CGO-based* code keeps allocating memory and does not free threads
>> properly:
>> package main
>>
>> import (
>> "fmt"
>> "runtime"
>> "runtime/debug"
>> "sync"
>> "time"
>> )
>>
>> /*
>> #include <unistd.h>
>> void cgoSleep() {
>> sleep(1);
>> }
>> */
>> import "C"
>>
>> func main() {
>> start := time.Now()
>>
>> var wg sync.WaitGroup
>> for i := 0; i < 5000; i++ {
>> wg.Add(1)
>> go func() {
>> defer wg.Done()
>> C.cgoSleep()
>> }()
>> }
>> wg.Wait()
>>
>> end := time.Now()
>>
>> // Force GC and free OS memory
>> runtime.GC()
>> debug.FreeOSMemory()
>> time.Sleep(10 * time.Second)
>>
>> var m runtime.MemStats
>> runtime.ReadMemStats(&m)
>>
>> fmt.Printf("Alloc = %v MiB", m.Alloc/1024/1024)
>> fmt.Printf("\tTotalAlloc = %v MiB", m.TotalAlloc/1024/1024)
>> fmt.Printf("\tSys = %v MiB", m.Sys/1024/1024)
>> fmt.Printf("\tNumGC = %v\n", m.NumGC)
>> fmt.Printf("Total time: %v\n", end.Sub(start))
>>
>> select {}
>> }
>>
>> *Expected Behavior*
>>
>> - The memory usage should *not* continue rising indefinitely.
>> - Threads should be properly cleaned up when they finish executing.
>> - The behavior should be similar to the following *pure Go*
>> equivalent, which does *not* exhibit the issue:
>>
>>
>>
>> *Actual Results* *With CGO (cgoSleep()):*
>>
>> - *Memory Usage:* *296 MB*
>> - *Threads:* *5,003*
>> - *System Memory (Sys from runtime.MemStats)*:
>>
>>
>> *205 MB *
>>
>> *With Pure Go (time.Sleep()):*
>>
>> - *Memory Usage:* *14 MB*
>> - *Threads:* *14*
>> - *System Memory (Sys from runtime.MemStats)*: *24 MB*
>>
>> *Additional Attempt*
>>
>> I tried forcing thread cleanup using runtime.LockOSThread() and
>> runtime.Goexit(), but *while the number of threads decreases, memory is
>> still never fully released*:
>> go func() { runtime.LockOSThread() defer wg.Done() C.cgoSleep()
>> runtime.Goexit() }() *Questions*
>>
>> 1. *Why is memory increasing indefinitely with CGO?*
>> 2. *Why are threads not getting properly cleaned up after CGO calls?*
>> 3. *Is there a way to force the Go runtime to reclaim memory
>> allocated for CGO threads?*
>> 4. *Is there a better approach to handling CGO calls that spawn
>> short-lived threads?*
>> 5. *Would using runtime.UnlockOSThread() help in this case, or is
>> this purely a CGO threading issue?*
>> 6. *Is there a way to track down where the memory is being held?
>> Since pprof does not show high memory usage, what other tools can I use?*
>>
>> *Go Version*
>> go1.23.5 darwin/arm64
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion visit
>> https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/golang-nuts/9a40383e-5b47-4f2b-bdea-a50946381650n%40googlegroups.com.
