[
https://issues.apache.org/jira/browse/KUDU-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859937#comment-16859937
]
Yingchun Lai commented on KUDU-2836:
------------------------------------
Thanks [~adar]
tcmalloc memory release logic seems not care MemTracker, it just count up to
128M to start GC, right?
In my situation, system memory 128G is willing to used up, Kudu used about
116G, and some other processes used few GiBs, and the Kudu process release
memory to OS very slowly, on memz page I see:
{code:java}
------------------------------------------------
MALLOC: 68494074240 (65321.0 MiB) Bytes in use by application
MALLOC: + 57308495872 (54653.6 MiB) Bytes in page heap freelist
MALLOC: + 974233808 ( 929.1 MiB) Bytes in central cache freelist
MALLOC: + 74752 ( 0.1 MiB) Bytes in transfer cache freelist
MALLOC: + 87554480 ( 83.5 MiB) Bytes in thread cache freelists
MALLOC: + 282722304 ( 269.6 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 127147155456 (121257.0 MiB) Actual memory used (physical + swap)
MALLOC: + 2475229184 ( 2360.6 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 129622384640 (123617.5 MiB) Virtual address space used
MALLOC:
MALLOC: 1930587 Spans in use
MALLOC: 728 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Total size of freelists for per-thread caches,
transfer cache, and central cache, by size class
------------------------------------------------
class 1 [ 8 bytes ] : 2396462 objs; 18.3 MiB; 18.3 cum MiB
class 2 [ 16 bytes ] : 250351 objs; 3.8 MiB; 22.1 cum MiB
class 3 [ 32 bytes ] : 1977119 objs; 60.3 MiB; 82.4 cum MiB
class 4 [ 48 bytes ] : 2982500 objs; 136.5 MiB; 219.0 cum MiB
class 5 [ 64 bytes ] : 845000 objs; 51.6 MiB; 270.5 cum MiB
class 6 [ 80 bytes ] : 1516178 objs; 115.7 MiB; 386.2 cum MiB
class 7 [ 96 bytes ] : 176661 objs; 16.2 MiB; 402.4 cum MiB
class 8 [ 112 bytes ] : 809641 objs; 86.5 MiB; 488.9 cum MiB
class 9 [ 128 bytes ] : 175100 objs; 21.4 MiB; 510.2 cum MiB
class 10 [ 144 bytes ] : 48859 objs; 6.7 MiB; 517.0 cum MiB
class 11 [ 160 bytes ] : 305376 objs; 46.6 MiB; 563.6 cum MiB
class 12 [ 176 bytes ] : 110207 objs; 18.5 MiB; 582.1 cum MiB
class 13 [ 192 bytes ] : 61828 objs; 11.3 MiB; 593.4 cum MiB
class 14 [ 208 bytes ] : 23991 objs; 4.8 MiB; 598.1 cum MiB
class 15 [ 224 bytes ] : 8716 objs; 1.9 MiB; 600.0 cum MiB
class 16 [ 240 bytes ] : 21531 objs; 4.9 MiB; 604.9 cum MiB
class 17 [ 256 bytes ] : 363291 objs; 88.7 MiB; 693.6 cum MiB
class 18 [ 288 bytes ] : 41100 objs; 11.3 MiB; 704.9 cum MiB
class 19 [ 320 bytes ] : 23715 objs; 7.2 MiB; 712.1 cum MiB
class 20 [ 352 bytes ] : 6195 objs; 2.1 MiB; 714.2 cum MiB
class 21 [ 384 bytes ] : 55479 objs; 20.3 MiB; 734.5 cum MiB
class 22 [ 416 bytes ] : 22092 objs; 8.8 MiB; 743.3 cum MiB
class 23 [ 448 bytes ] : 11544 objs; 4.9 MiB; 748.2 cum MiB
class 24 [ 480 bytes ] : 8437 objs; 3.9 MiB; 752.1 cum MiB
class 25 [ 512 bytes ] : 156958 objs; 76.6 MiB; 828.7 cum MiB
class 26 [ 576 bytes ] : 5235 objs; 2.9 MiB; 831.6 cum MiB
class 27 [ 640 bytes ] : 9703 objs; 5.9 MiB; 837.5 cum MiB
class 28 [ 704 bytes ] : 2789 objs; 1.9 MiB; 839.4 cum MiB
class 29 [ 768 bytes ] : 5589 objs; 4.1 MiB; 843.5 cum MiB
class 30 [ 896 bytes ] : 4310 objs; 3.7 MiB; 847.2 cum MiB
class 31 [ 1024 bytes ] : 11443 objs; 11.2 MiB; 858.4 cum MiB
class 32 [ 1152 bytes ] : 3784 objs; 4.2 MiB; 862.5 cum MiB
class 33 [ 1280 bytes ] : 2489 objs; 3.0 MiB; 865.6 cum MiB
class 34 [ 1408 bytes ] : 3460 objs; 4.6 MiB; 870.2 cum MiB
class 35 [ 1536 bytes ] : 2690 objs; 3.9 MiB; 874.1 cum MiB
class 36 [ 1792 bytes ] : 6256 objs; 10.7 MiB; 884.8 cum MiB
class 37 [ 2048 bytes ] : 4346 objs; 8.5 MiB; 893.3 cum MiB
class 38 [ 2304 bytes ] : 1502 objs; 3.3 MiB; 896.6 cum MiB
class 39 [ 2560 bytes ] : 1350 objs; 3.3 MiB; 899.9 cum MiB
class 40 [ 2816 bytes ] : 1322 objs; 3.6 MiB; 903.5 cum MiB
class 41 [ 3072 bytes ] : 681 objs; 2.0 MiB; 905.5 cum MiB
class 42 [ 3328 bytes ] : 615 objs; 2.0 MiB; 907.4 cum MiB
class 43 [ 4096 bytes ] : 1054 objs; 4.1 MiB; 911.5 cum MiB
class 44 [ 4608 bytes ] : 421 objs; 1.9 MiB; 913.4 cum MiB
class 45 [ 5120 bytes ] : 805 objs; 3.9 MiB; 917.3 cum MiB
class 46 [ 6144 bytes ] : 882 objs; 5.2 MiB; 922.5 cum MiB
class 47 [ 6656 bytes ] : 452 objs; 2.9 MiB; 925.3 cum MiB
class 48 [ 8192 bytes ] : 556 objs; 4.3 MiB; 929.7 cum MiB
class 49 [ 9216 bytes ] : 547 objs; 4.8 MiB; 934.5 cum MiB
class 50 [ 10240 bytes ] : 75 objs; 0.7 MiB; 935.2 cum MiB
class 51 [ 12288 bytes ] : 162 objs; 1.9 MiB; 937.1 cum MiB
class 52 [ 13312 bytes ] : 20 objs; 0.3 MiB; 937.4 cum MiB
class 53 [ 16384 bytes ] : 8 objs; 0.1 MiB; 937.5 cum MiB
class 54 [ 20480 bytes ] : 794 objs; 15.5 MiB; 953.0 cum MiB
class 55 [ 24576 bytes ] : 2 objs; 0.0 MiB; 953.1 cum MiB
class 56 [ 26624 bytes ] : 5 objs; 0.1 MiB; 953.2 cum MiB
class 57 [ 32768 bytes ] : 85 objs; 2.7 MiB; 955.8 cum MiB
class 58 [ 40960 bytes ] : 20 objs; 0.8 MiB; 956.6 cum MiB
class 59 [ 49152 bytes ] : 3 objs; 0.1 MiB; 956.8 cum MiB
class 60 [ 57344 bytes ] : 5 objs; 0.3 MiB; 957.0 cum MiB
class 61 [ 65536 bytes ] : 14 objs; 0.9 MiB; 957.9 cum MiB
class 62 [ 73728 bytes ] : 12 objs; 0.8 MiB; 958.8 cum MiB
class 64 [ 90112 bytes ] : 1 objs; 0.1 MiB; 958.8 cum MiB
class 67 [ 114688 bytes ] : 1 objs; 0.1 MiB; 959.0 cum MiB
class 68 [ 122880 bytes ] : 2 objs; 0.2 MiB; 959.2 cum MiB
class 77 [ 196608 bytes ] : 2 objs; 0.4 MiB; 959.6 cum MiB
class 85 [ 262144 bytes ] : 2 objs; 0.5 MiB; 960.1 cum MiB
------------------------------------------------
PageHeap: 69 sizes; 54653.6 MiB free; 2360.6 MiB unmapped
------------------------------------------------
1 pages * 26403 spans ~ 206.3 MiB; 206.3 MiB cum; unmapped: 19.8 MiB;
19.8 MiB cum
2 pages * 55083 spans ~ 860.7 MiB; 1066.9 MiB cum; unmapped: 598.6 MiB;
618.4 MiB cum
3 pages * 52777 spans ~ 1237.0 MiB; 2303.9 MiB cum; unmapped: 375.4 MiB;
993.8 MiB cum
4 pages * 56007 spans ~ 1750.2 MiB; 4054.1 MiB cum; unmapped: 1056.1 MiB;
2049.8 MiB cum
5 pages * 67844 spans ~ 2650.2 MiB; 6704.3 MiB cum; unmapped: 0.7 MiB;
2050.5 MiB cum
6 pages * 322 spans ~ 15.1 MiB; 6719.4 MiB cum; unmapped: 0.0 MiB;
2050.5 MiB cum
7 pages * 2854 spans ~ 156.1 MiB; 6875.5 MiB cum; unmapped: 0.7 MiB;
2051.3 MiB cum
8 pages * 6838 spans ~ 427.4 MiB; 7302.8 MiB cum; unmapped: 0.8 MiB;
2052.0 MiB cum
9 pages * 3640 spans ~ 255.9 MiB; 7558.8 MiB cum; unmapped: 0.9 MiB;
2052.9 MiB cum
10 pages * 11784 spans ~ 920.6 MiB; 8479.4 MiB cum; unmapped: 1.1 MiB;
2054.0 MiB cum
11 pages * 1147 spans ~ 98.6 MiB; 8578.0 MiB cum; unmapped: 1.3 MiB;
2055.3 MiB cum
12 pages * 780 spans ~ 73.1 MiB; 8651.1 MiB cum; unmapped: 1.8 MiB;
2057.1 MiB cum
13 pages * 1660 spans ~ 168.6 MiB; 8819.7 MiB cum; unmapped: 1.8 MiB;
2058.9 MiB cum
14 pages * 813 spans ~ 88.9 MiB; 8908.6 MiB cum; unmapped: 2.2 MiB;
2061.1 MiB cum
15 pages * 7671 spans ~ 898.9 MiB; 9807.5 MiB cum; unmapped: 2.3 MiB;
2063.4 MiB cum
16 pages * 843 spans ~ 105.4 MiB; 9912.9 MiB cum; unmapped: 1.9 MiB;
2065.3 MiB cum
17 pages * 814 spans ~ 108.1 MiB; 10021.0 MiB cum; unmapped: 3.1
MiB; 2068.4 MiB cum
18 pages * 2771 spans ~ 389.7 MiB; 10410.7 MiB cum; unmapped: 3.0
MiB; 2071.3 MiB cum
19 pages * 1133 spans ~ 168.2 MiB; 10578.9 MiB cum; unmapped: 2.7
MiB; 2074.0 MiB cum
20 pages * 7063 spans ~ 1103.6 MiB; 11682.5 MiB cum; unmapped: 2.2
MiB; 2076.2 MiB cum
21 pages * 711 spans ~ 116.6 MiB; 11799.1 MiB cum; unmapped: 3.1
MiB; 2079.3 MiB cum
22 pages * 469 spans ~ 80.6 MiB; 11879.7 MiB cum; unmapped: 4.6
MiB; 2083.9 MiB cum
23 pages * 942 spans ~ 169.3 MiB; 12049.0 MiB cum; unmapped: 3.2
MiB; 2087.2 MiB cum
24 pages * 545 spans ~ 102.2 MiB; 12151.2 MiB cum; unmapped: 3.0
MiB; 2090.2 MiB cum
25 pages * 9395 spans ~ 1835.0 MiB; 13986.1 MiB cum; unmapped: 2.7
MiB; 2092.9 MiB cum
26 pages * 1078 spans ~ 219.0 MiB; 14205.1 MiB cum; unmapped: 3.2
MiB; 2096.2 MiB cum
27 pages * 484 spans ~ 102.1 MiB; 14307.2 MiB cum; unmapped: 4.0
MiB; 2100.2 MiB cum
28 pages * 1813 spans ~ 396.6 MiB; 14703.8 MiB cum; unmapped: 3.3
MiB; 2103.5 MiB cum
29 pages * 1962 spans ~ 444.5 MiB; 15148.3 MiB cum; unmapped: 4.8
MiB; 2108.2 MiB cum
30 pages * 19933 spans ~ 4671.8 MiB; 19820.1 MiB cum; unmapped: 4.2
MiB; 2112.4 MiB cum
31 pages * 2201 spans ~ 533.1 MiB; 20353.2 MiB cum; unmapped: 4.1
MiB; 2116.5 MiB cum
32 pages * 1747 spans ~ 436.8 MiB; 20789.9 MiB cum; unmapped: 6.0
MiB; 2122.5 MiB cum
33 pages * 1 spans ~ 0.3 MiB; 20790.2 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
34 pages * 1 spans ~ 0.3 MiB; 20790.4 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
35 pages * 2 spans ~ 0.5 MiB; 20791.0 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
36 pages * 1 spans ~ 0.3 MiB; 20791.3 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
38 pages * 1 spans ~ 0.3 MiB; 20791.6 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
40 pages * 2 spans ~ 0.6 MiB; 20792.2 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
62 pages * 3 spans ~ 1.5 MiB; 20793.6 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
63 pages * 1 spans ~ 0.5 MiB; 20794.1 MiB cum; unmapped: 0.0
MiB; 2122.5 MiB cum
100 pages * 85 spans ~ 66.4 MiB; 20860.5 MiB cum; unmapped: 6.2
MiB; 2128.8 MiB cum
101 pages * 105 spans ~ 82.9 MiB; 20943.4 MiB cum; unmapped: 10.3
MiB; 2139.1 MiB cum
102 pages * 70 spans ~ 55.8 MiB; 20999.2 MiB cum; unmapped: 8.8
MiB; 2147.8 MiB cum
103 pages * 79 spans ~ 63.6 MiB; 21062.8 MiB cum; unmapped: 8.0
MiB; 2155.9 MiB cum
104 pages * 528 spans ~ 429.0 MiB; 21491.8 MiB cum; unmapped: 3.2
MiB; 2159.1 MiB cum
105 pages * 81 spans ~ 66.4 MiB; 21558.2 MiB cum; unmapped: 6.6
MiB; 2165.7 MiB cum
106 pages * 75 spans ~ 62.1 MiB; 21620.3 MiB cum; unmapped: 10.8
MiB; 2176.4 MiB cum
107 pages * 59 spans ~ 49.3 MiB; 21669.6 MiB cum; unmapped: 9.2
MiB; 2185.6 MiB cum
108 pages * 73 spans ~ 61.6 MiB; 21731.2 MiB cum; unmapped: 11.0
MiB; 2196.6 MiB cum
109 pages * 272 spans ~ 231.6 MiB; 21962.8 MiB cum; unmapped: 5.1
MiB; 2201.7 MiB cum
110 pages * 68 spans ~ 58.4 MiB; 22021.3 MiB cum; unmapped: 5.2
MiB; 2206.9 MiB cum
111 pages * 47 spans ~ 40.8 MiB; 22062.0 MiB cum; unmapped: 11.3
MiB; 2218.1 MiB cum
112 pages * 53 spans ~ 46.4 MiB; 22108.4 MiB cum; unmapped: 11.4
MiB; 2229.5 MiB cum
113 pages * 41 spans ~ 36.2 MiB; 22144.6 MiB cum; unmapped: 11.5
MiB; 2241.0 MiB cum
114 pages * 198 spans ~ 176.3 MiB; 22321.0 MiB cum; unmapped: 4.5
MiB; 2245.5 MiB cum
115 pages * 32 spans ~ 28.8 MiB; 22349.7 MiB cum; unmapped: 5.4
MiB; 2250.8 MiB cum
116 pages * 28 spans ~ 25.4 MiB; 22375.1 MiB cum; unmapped: 10.0
MiB; 2260.8 MiB cum
117 pages * 24 spans ~ 21.9 MiB; 22397.0 MiB cum; unmapped: 6.4
MiB; 2267.2 MiB cum
118 pages * 38 spans ~ 35.0 MiB; 22432.0 MiB cum; unmapped: 11.1
MiB; 2278.3 MiB cum
119 pages * 127 spans ~ 118.1 MiB; 22550.1 MiB cum; unmapped: 4.6
MiB; 2282.9 MiB cum
120 pages * 32 spans ~ 30.0 MiB; 22580.1 MiB cum; unmapped: 7.5
MiB; 2290.4 MiB cum
121 pages * 18 spans ~ 17.0 MiB; 22597.1 MiB cum; unmapped: 13.2
MiB; 2303.7 MiB cum
122 pages * 29 spans ~ 27.6 MiB; 22624.8 MiB cum; unmapped: 10.5
MiB; 2314.1 MiB cum
123 pages * 28 spans ~ 26.9 MiB; 22651.7 MiB cum; unmapped: 16.3
MiB; 2330.5 MiB cum
124 pages * 208 spans ~ 201.5 MiB; 22853.2 MiB cum; unmapped: 6.8
MiB; 2337.3 MiB cum
125 pages * 39 spans ~ 38.1 MiB; 22891.3 MiB cum; unmapped: 8.8
MiB; 2346.0 MiB cum
126 pages * 44 spans ~ 43.3 MiB; 22934.6 MiB cum; unmapped: 10.8
MiB; 2356.9 MiB cum
127 pages * 159 spans ~ 157.8 MiB; 23092.3 MiB cum; unmapped: 1.0
MiB; 2357.9 MiB cum
128 pages * 3 spans ~ 3.0 MiB; 23095.3 MiB cum; unmapped: 0.0
MiB; 2357.9 MiB cum
>128 large * 8310 spans ~ 33918.9 MiB; 57014.2 MiB cum; unmapped: 2.7
>MiB; 2360.6 MiB cum
{code}
"Total consumption" memory is about 65G, and
tcmalloc_max_free_bytes_percentage=10, does it mean that 57308495872(Bytes in
page heap freelist) - 65G*10% can be returned? I read code of GcTcmalloc(), but
not sure.
> Maybe wrong memory size used to detect pressure
> -----------------------------------------------
>
> Key: KUDU-2836
> URL: https://issues.apache.org/jira/browse/KUDU-2836
> Project: Kudu
> Issue Type: Improvement
> Components: tserver
> Reporter: Yingchun Lai
> Assignee: Yingchun Lai
> Priority: Critical
>
> One of my tserver, totally 128G memory, gflags:
> {code:java}
> -memory_limit_hard_bytes=107374182475 (100G)
> -memory_limit_soft_percentage=85 -memory_pressure_percentage=80{code}
> Memory used about 95%, "top" result like:
> {code:java}
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 8359 work 20 0 0.326t 0.116t 81780 S 727.9 94.6 230228:10 kudu_tablet_ser
> {code}
> That is kudu_tablet_server process used about 116G memory.
> On mem-trackers page, I find the "Total consumption" value is about 65G, much
> lower than 116G.
> Then, I login to the server and read code to check any free memory MM
> operations are work correctly. Unfortunatly, the memory pressure detect
> function(process_memory::UnderMemoryPressure) doesn't report it's under
> pressure, because the tcmalloc function GetNumericProperty(const char*
> property, size_t* value) with parameter "generic.current_allocated_bytes"
> doesn't return the memory as the memory use reported by the OS.
> [https://gperftools.github.io/gperftools/tcmalloc.html]
> {quote}
> |{{generic.current_allocated_bytes}}|Number of bytes used by the application.
> This will not typically match the memory use reported by the OS, because it
> does not include TCMalloc overhead or memory fragmentation.|
> {quote}
> This situation may lead to OPs prefer to free memory could not be scheduled
> promptly, and the OS memory may consumed empty, and then kill tserver because
> of OOM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
