Leaving the deployment running for a while after the 3rd restart of the target - 6 rounds of WAL truncation, the memory goes up to 3.7Gi comparing to 2.5Gi before doing the restart. There must be something that Prometheus holds back for this upgrade/restart scenario I guess.
[image: ask_prometheus_user_memory_increase_after_target_restart_upgrade_v2.jpg] On Tuesday, October 31, 2023 at 10:07:24 PM UTC+7 Vu Nguyen wrote: > We have Prometheus v2.47.1 deployed on k8s; scraping 500k time series from > a single target (*) > > When restart the target, the number of time series in HEAD block jump to > 1M (1), and Prometheus memory increases from the average of 2.5Gi to 3Gi. > Leave Prometheus running for few WAL truncation cycles, the memory still > not go back to the point before restarting the target even the number of > time series in HEAD block back to 500K. > > If I trigger another target restarts, that memory keeps going up. Here is > the graph: > > Could you please help us understand why the memory does not fallback to > the initial point (*) before we restart/upgrade target? > > [1] k8s pod restart will come up with a new IP - new instance label value; > therefore, a new set of of 500K time series is generated. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/eca095a3-ac30-42f4-b666-13f6a520f59bn%40googlegroups.com.
