Thanks for quick response. Here is the complete scenario Prometheus 2.45 version is used and remote writes to Thanos. Prometheus is run in agent mode. Thanos is hosted in EKS as a HA. Thanos 0.32.5 is used. Thanos receiver is setup to accept metrics 24hrs into past and 6hrs into future using following flags: --tsdb.too-far-in-future.time-window=6h --tsdb.out-of-order.time-window=24h
All times are in UTC. Say that correct time is 10:00AM. We have a system that is out of time sync with no NTP configured and is at 09:00AM. We have windows exporter to get OS metrics. We have a customer exporter written in GoLang that scrapes metrics from windows exporter and corrects the time to current UTC Time. This custom exporter is scrapped by Prometheus for every 4 minutes. The custom exporter is written in GoLang and uses NewMetricWithTimestamp to add the correct timestamp. Following are the observations: If we scrape windows exporter directly from Prometheus and remote write to Thanos, Thanos accepts the metrics as Out of order ingestion is allowed. If we scrape our custom exporter then we see metrics too old or too future error in Prometheus logs On Monday, December 18, 2023 at 5:43:39 PM UTC+5:30 Brian Candler wrote: > On Monday 18 December 2023 at 06:41:41 UTC Kesavanand Chavali wrote: > > We have a node where time is not in sync. we implemented an exporter in Go > that will scrape the metrics from other exporters and corrects the > timestamp. Prometheus scrapes this exporter but errors out with the above > said message. > > > I don't understand the problem. > > If the node where the exporter is running has a bad clock, it shouldn't > make any difference. Firstly, the exported metrics generally won't have a > timestamp (it's an optional field). Secondly, even if they do, Prometheus > ignores it anyway unless you set "honor_timestamps: true", which is almost > always the wrong thing to do. > > Do you mean you are using the remote_write protocol with the remote_write > receiver, rather than scraping? > > Perhaps you can sketch a diagram of what the components are, and how they > are talking to each other. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0720d07c-92de-496c-ac61-f99441a2b70an%40googlegroups.com.
