[go-nuts] Why is there a huge gap between the memory profile's inuse_space in pprof and the externally obtained memory footprint?

[Zxilly Chou]

I'm working on the project https://github.com/Zxilly/go-size-analyzer, a 
program that can be very memory intensive, so I've written some scripts to 
fetch the pprof file during integration testing, and I'm using python's 
psutil to read the cpu and memory usage to automatically plot into the 
images.

Recently I was working on some optimizations so I started to analyze the 
pprof files collected in CI, my test program prints the heap profile file 
once per second, but I noticed that in all mem pprof files the process 
memory footprint varies dramatically for both inuse_space and psutil reads. 
Here are two images showing the data read by the same test.



As shown in the figure, the difference in size between the memory read by 
psutil and the memory fetched by pprof is almost 800mb. I know that mem 
pprof can't count data on the stack, but I don't think the stack holds that 
much.

You can download the data collected by CI and the rendered images at 
https://github.com/Zxilly/go-size-analyzer/actions/runs/10007859171/artifacts/1718981276.
 
The data is in the cockroach-linux-amd64/json directory in the tarball.

Below is the key code for collecting data.


python part:

    ps_process = psutil.Process(process.pid)
    
    while process.poll() is None:
        percent = ps_process.cpu_percent(interval=0.1)
        mem = ps_process.memory_info().rss / (1024 * 1024)
        elapsed_time = time.time() - start_time
        if elapsed_time > timeout:
            raise TimeoutError(f"Process {name} timed out after {timeout} 
seconds.")
    
        if draw:
            cpu_percentages.append(percent)
            memory_usage_mb.append(mem)
            timestamps.append(elapsed_time)


golang part:

    var ctx context.Context
    ctx, heapProfileStop = context.WithCancel(context.Background())
    
    go func() {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
    
        id := 0
        write := func() {
           id++
           path := filepath.Join(outputDir, fmt.Sprintf("mem-%d.pprof", id))
           f, err := os.Create(path)
           defer func(f *os.File) {
              err = f.Close()
              if err != nil {
                 panic(err)
              }
           }(f)
           if err != nil {
              panic(err)
           }
    
           err = pprof.WriteHeapProfile(f)
           if err != nil {
              panic(err)
           }
        }
    
        for {
           select {
           case <-ctx.Done():
              write()
              return
           case <-ticker.C:
              write()
           }
        }
    }()

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/117b16d4-a1a2-46a1-915e-8b7d8b607a46n%40googlegroups.com.