I could see the number of time series are truncated to a half, from 1M to 500K but the memory does not go back to the point before target got restarted (1)
If I keep restarting the target the second time, the number of time series will jump up to 1M and will eventually drop to 500M after few HEAD truncation cycles but the memory now could be even higher than the case 1 above. What confusing me here is that Prometheus memory does not return to the point before restarting the target even the time series in HEAD goes back to 500K. On Thu, Nov 30, 2023 at 6:16 PM Bryan Boreham <[email protected]> wrote: > > > > On 30 Nov 2023, at 07:20, Vu Nguyen <[email protected]> wrote: > > > > > > Thank you very much for your supports. > > > > > are you still deleting the WAL? > > > > No, I did not delete WAL at all. What I did was restarting a pod that > has 500K time series exposed. > > > > Ah, if there is any label different (eg the pod name) then that creates > new series. > So this would be consistent with the number of series bouncing up to > 1,000,000. > Prometheus only clears stale series out of the head after compaction, so > you see series and memory go up for a couple of hours then come down again. > > In other words this is standard behaviour. > > Bryan > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAF7KRaxGCRsfoW7Z7hTKwwx_vZzx9UddS9cKm-OztZG%3Duz90vw%40mail.gmail.com.
