Hi Stuart, I can see that support of timestamps has been discontinued around November 21st, 2021. Indeed, when I try
C02G74F9Q6LR bat-datapipeline % echo "test_metric_with_timestamp 33 1665623039" | curl --data-binary @- https://<pushgatewayURL>/metrics/job/pushgateway-job I get *"pushed metrics are invalid or inconsistent with existing metrics: pushed metrics must not have timestamps"* Could you please specify how do you use timestamps in metrics? Thanks Le samedi 18 juin 2022 à 13:13:47 UTC+2, Stuart Clark a écrit : > On 14/06/2022 18:32, Jeremy Collette wrote: > > Hello, > > We have written a custom exporter that exposes metrics with explicit > timestamps, which Prometheus periodically scrapes. In the case where > Prometheus becomes temporarily unavailable, these metric samples will be > cached in the exporter until they are scraped, causing affected metrics to > age. > > I understand that if a metric is older than a certain threshold, it will > be rejected by Prometheus with the message: "Error on ingesting samples > that are too old or are too far into the future". > > I'm trying to understand if there are any guarantees surrounding the > ingestion of historical metrics. Is there some metric sample age that is > guaranteed to be recent enough to be ingested? For example, are samples > with timestamps within the last hour always going to be considered recent? > Within the last five minutes? > > According to this previous thread: Error on ingesting samples that are > too old > <https://groups.google.com/g/prometheus-users/c/rKJYm6naEow/m/zylud_J4AAAJ>, > MR seems to indicate that metrics as old as 1 second can be dropped due to > being too old. Is this interpretation correct? If so, is there any way to > ensure metrics with timestamps won't be dropped for being too old? > > The use of timestamps in metrics is not something that should be used > except in some very specific cases. The main use case for adding a > timestamp is when you are scraping metrics into Prometheus that have been > sourced from another existing metrics system (for example things like the > Cloudwatch Exporter). You also mention something about your exporter > caching things until they are scraped, which also sounds like something > that is not advisable. The action of the exporter shouldn't really be > changing depending on the requests being received (or not received). > > An exporter is expected to return the various metrics that reflect "now", > in the same way that a directly instrumented application would be expected > to return the current state of the metrics being maintained in memory. For > a simple exporter the normal mechanism is for a request to be received > which then triggers some mechanism to generate the metrics. For example > with something like the MySQL Exporter a request would trigger a query on > the connected database which then returns various information that is > converted into Prometheus metrics and returned. In some situations the > process to fetch information from the underlying system can be quite > resource intensive or slow. In that case a common design is to decouple the > information fetching process from the request handling process. One example > is to perform the information fetching process on a periodic timer, with > the information fetched then stored in memory. The request process then > reads and returns that information - returning the same values for every > request until the next cycle of the information fetching process. In none > of these standard scenarios would you expect timestamps to be attached to > the returned metrics. > > It would be good to hear a bit more about what you are trying to do, as it > is highly likely that the use of timestamps in your use case is probably > not the right option and they should just be dropped. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6b8f9d42-40e7-41db-856f-94e292db8b04n%40googlegroups.com.
