[go-nuts] Re: Why is there a huge gap between the memory profile's inuse_space in pprof and the externally obtained memory footprint?

[王旭东]

Hi,
I'm not an expert on pprof (Go) or psutil (Python).
But I find this page 
<https://stackoverflow.com/questions/73632745/memory-usage-discrepency-between-pprof-and-ps>
 might 
be helpful.

Thanks,
Xudong

在2024年7月20日星期六 UTC+8 15:21:05<Zxilly Chou> 写道：

> I'm working on the project https://github.com/Zxilly/go-size-analyzer, a 
> program that can be very memory intensive, so I've written some scripts to 
> fetch the pprof file during integration testing, and I'm using python's 
> psutil to read the cpu and memory usage to automatically plot into the 
> images.
>
> Recently I was working on some optimizations so I started to analyze the 
> pprof files collected in CI, my test program prints the heap profile file 
> once per second, but I noticed that in all mem pprof files the process 
> memory footprint varies dramatically for both inuse_space and psutil reads. 
> Here are two images showing the data read by the same test.
>
>
>
> As shown in the figure, the difference in size between the memory read by 
> psutil and the memory fetched by pprof is almost 800mb. I know that mem 
> pprof can't count data on the stack, but I don't think the stack holds that 
> much.
>
> You can download the data collected by CI and the rendered images at 
> https://github.com/Zxilly/go-size-analyzer/actions/runs/10007859171/artifacts/1718981276.
>  
> The data is in the cockroach-linux-amd64/json directory in the tarball.
>
> Below is the key code for collecting data.
>
>
> python part:
>
>     ps_process = psutil.Process(process.pid)
>     
>     while process.poll() is None:
>         percent = ps_process.cpu_percent(interval=0.1)
>         mem = ps_process.memory_info().rss / (1024 * 1024)
>         elapsed_time = time.time() - start_time
>         if elapsed_time > timeout:
>             raise TimeoutError(f"Process {name} timed out after {timeout} 
> seconds.")
>     
>         if draw:
>             cpu_percentages.append(percent)
>             memory_usage_mb.append(mem)
>             timestamps.append(elapsed_time)
>
>
> golang part:
>
>     var ctx context.Context
>     ctx, heapProfileStop = context.WithCancel(context.Background())
>     
>     go func() {
>         ticker := time.NewTicker(1 * time.Second)
>         defer ticker.Stop()
>     
>         id := 0
>         write := func() {
>            id++
>            path := filepath.Join(outputDir, fmt.Sprintf("mem-%d.pprof", 
> id))
>            f, err := os.Create(path)
>            defer func(f *os.File) {
>               err = f.Close()
>               if err != nil {
>                  panic(err)
>               }
>            }(f)
>            if err != nil {
>               panic(err)
>            }
>     
>            err = pprof.WriteHeapProfile(f)
>            if err != nil {
>               panic(err)
>            }
>         }
>     
>         for {
>            select {
>            case <-ctx.Done():
>               write()
>               return
>            case <-ticker.C:
>               write()
>            }
>         }
>     }()
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e8df9a0a-1d7c-421f-9943-8675243254b2n%40googlegroups.com.