That only shows an in-use memory of 573MB.
https://pprof.me/248ce153c305ede009283fef7d9fd15e/?profileType=profile%3Ainuse_space%3Abytes%3Aspace%3Abytes
Can you post a graph of process_resident_memory_bytes{job="prometheus"}?
What metric are you using to report?
On Thu, Nov 7, 2024 at 10:43 AM Bhanu Prakash <bhanuprakash8...@gmail.com>
wrote:
> Hi ben,
>
> I have upload the heap file below is the link.
>
> https://pprof.me/248ce153c305ede009283fef7d9fd15e/?profileType=profile%253Aalloc_objects%253Acount%253Aspace%253Abytes&color_by=filename
>
> *and tsdb-status*
>
> curl http://10.224.0.122:9090/api/v1/status/tsdb
>
> {"status":"success","data":{"headStats":{"numSeries":162587,"numLabelPairs":8046,"chunkCount":199721,"minTime":1730966400000,"maxTime":1730971
>
> 996605},"seriesCountByMetricName":[{"name":"apiserver_request_duration_seconds_bucket","value":6624},{"name":"etcd_request_duration_seconds_bu
>
> cket","value":5304},{"name":"apiserver_request_sli_duration_seconds_bucket","value":4400},{"name":"container_tasks_state","value":3280},{"name
>
> ":"kube_pod_status_reason","value":2930},{"name":"kube_pod_status_phase","value":2930},{"name":"container_memory_failures_total","value":2624}
>
> ,{"name":"kubelet_runtime_operations_duration_seconds_bucket","value":2235},{"name":"apiserver_request_body_size_bytes_bucket","value":2208},{
>
> "name":"kube_replicaset_status_observed_generation","value":2060}],"labelValueCountByLabelName":[{"name":"__name__","value":1250},{"name":"nam
>
> e","value":692},{"name":"id","value":608},{"name":"replicaset","value":517},{"name":"mountpoint","value":450},{"name":"lease","value":310},{"n
>
> ame":"lease_holder","value":309},{"name":"uid","value":284},{"name":"container_id","value":272},{"name":"le","value":252}],"memoryInBytesByLab
>
> elName":[{"name":"id","value":100200},{"name":"mountpoint","value":52087},{"name":"__name__","value":44903},{"name":"name","value":33040},{"na
>
> me":"container_id","value":20944},{"name":"replicaset","value":15060},{"name":"lease_holder","value":10996},{"name":"lease","value":10506},{"n
>
> ame":"uid","value":10224},{"name":"image_id","value":9545}],"seriesCountByLabelValuePair":[{"name":"app_kubernetes_io_managed_by=Helm","value"
>
> :78324},{"name":"app_kubernetes_io_instance=loki-stack","value":78324},{"name":"job=kubernetes-service-endpoints","value":78324},{"name":"app_
>
> kubernetes_io_component=metrics","value":78324},{"name":"app_kubernetes_io_part_of=kube-state-metrics","value":68200},{"name":"app_kubernetes_
>
> io_version=2.8.0","value":68200},{"name":"helm_sh_chart=kube-state-metrics-4.30.0","value":68200},{"name":"app_kubernetes_io_name=kube-state-m
>
> etrics","value":68200},{"name":"service=loki-stack-kube-state-metrics","value":66492},{"name":"node=aks-grafana-14132910-vmss000025","value":6
> 6269}]}}
>
> Thanks,
> Bhanu
> On Thursday, November 7, 2024 at 2:27:48 PM UTC+5:30 Ben Kochie wrote:
>
>> I would recommend getting a heap snapshot and posting it to
>> https://pprof.me.
>>
>> curl -o heap.pprof http://localhost:9090/debug/pprof/heap
>>
>> Also including http://localhost:9090/tsdb-status would help.
>>
>> On Thu, Nov 7, 2024 at 9:53 AM Bhanu Prakash <bhanupra...@gmail.com>
>> wrote:
>>
>>> Hello Prometheus Community,
>>>
>>> I’m encountering a memory usage issue with my Prometheus server,
>>> particularly during and after startup, and I’m hoping to get some insights
>>> on optimizing it.
>>>
>>> *Problem:* Upon startup, my Prometheus instance consumes a large amount
>>> of memory, primarily due to the WAL (Write-Ahead Log) replay. To address
>>> this, I enabled --enable-feature=memory-snapshot-on-shutdown, expecting it
>>> to reduce the startup memory spike by eliminating the need for full WAL
>>> replay. However, I’m still seeing the memory usage spike to around 5GB on
>>> startup. Once started, Prometheus continues to hold this high memory,
>>> without releasing it back to the system.
>>>
>>> Is there a recommended way to configure Prometheus to release memory
>>> post-startup?
>>> Are there additional configurations or optimizations for large WAL files
>>> or memory management that could help?
>>>
>>> Any guidance or suggestions would be greatly appreciated!
>>>
>>> Thank you,
>>> BhanuPrakash.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion visit
>>> https://groups.google.com/d/msgid/prometheus-users/99e6725f-f757-45f4-80a4-98c60e2b5063n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/prometheus-users/99e6725f-f757-45f4-80a4-98c60e2b5063n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/prometheus-users/1dbeaf12-c074-45da-8679-c2370e150861n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/1dbeaf12-c074-45da-8679-c2370e150861n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/CABbyFmpVq8Dgq7hpVennpvyGd%3DH7_3CvVWJBntpShow2xcEV0g%40mail.gmail.com.
