I have several basic questions. I examined a couple of golang programs,
and got really confused.
I started by looking at a kube-proxy from Kubernetes release 1.15 (probably
either 1.15.9 or 1.15.12) on Linux 4.15.0 using `/proc/[pid]/statm`.
As a reminder, here is what man procfs has to say about that:
Provides information about memory usage, measured in pages. The
columns are:
size (1) total program size
(same as VmSize in /proc/[pid]/status)
resident (2) resident set size
(same as VmRSS in /proc/[pid]/status)
shared (3) number of resident shared pages (i.e., backed
by a file)
(same as RssFile+RssShmem in /proc/[pid]/status)
text (4) text (code)
lib (5) library (unused since Linux 2.6; always 0)
data (6) data + stack
dt (7) dirty pages (unused since Linux 2.6; always 0)
Here is what I saw:
/proc/10368$ cat statm
35207 8684 6285 4026 0 26265 0
The page size is 4 KiB. That’s about 16 MB of code plus 108 MB of
data+stack, plus another 20 MB of something else. What is that something
else? Why is the data+stack so huge?
Next I looked at `maps`:
/proc/10368$ sudo cat maps
00400000-013ba000 r-xp 00000000 fc:01 777880
/usr/local/bin/kube-proxy
013ba000-026e9000 r--p 00fba000 fc:01 777880
/usr/local/bin/kube-proxy
026e9000-02749000 rw-p 022e9000 fc:01 777880
/usr/local/bin/kube-proxy
02749000-02770000 rw-p 00000000 00:00 0
c000000000-c000200000 rw-p 00000000 00:00 0
c000200000-c000400000 rw-p 00000000 00:00 0
c000400000-c000a00000 rw-p 00000000 00:00 0
c000a00000-c004000000 rw-p 00000000 00:00 0
7f4b47346000-7f4b49937000 rw-p 00000000 00:00 0
7ffcd6169000-7ffcd618a000 rw-p 00000000 00:00 0 [stack]
7ffcd61d0000-7ffcd61d3000 r--p 00000000 00:00 0 [vvar]
7ffcd61d3000-7ffcd61d5000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
The block from c000000000 to c004000000 is about 67 MB, and the block from
7f4b47346000 to 7f4b49937000 is about another 40 MB. So that must be most
of the 108 MB data+stack. The fact that they are distinct memory ranges
suggest they are two different kinds of usage. Which range is which usage?
The first block for /usr/local/bin/kube-proxy is the 4026 pages that procfs
reports for text. Why are the other two /usr/local/bin/kube-proxy blocks
not reported as text? They add up to about that mysterious 20 MB. Is the
first block included in "number of resident shared pages (i.e., backed by a
file)"?
Next, I tried scraping /metrics from a different golang process. I have
lots of questions about the memory related metrics. Is there any more
definition of the go_memstats_… metrics beyond their HELP lines?
For example, what is the difference beween
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in
use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.881872e+06
and
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 7.33184e+06
?
It clearly is not
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.885952e+07
… and what is that, anyway?
And then there is this:
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.7155584e+07
… is that “released” as in “cumulative over release operations in the past”
or as in “currently in released-to-os state”?
Same question for “obtained” in
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.619136e+07
Then there is
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack
allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 917504
does “use by the stack allocator” refer to overhead of the allocator, or
use for stacks themselves?
There are several metrics described as being “obtained from system” for
various purposes, but they do not add up to what appears to be the total.
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.619136e+07
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures
obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures
obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 131072
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for
stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 917504
Those add up to about 67256320, but then there is
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1762168e+07
Thanks,
Mike
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/2b3242a1-e752-4e3b-bcf0-c32d6c1820a6o%40googlegroups.com.
