Hi Brian, Thanks for your input, it as always apreaciated. I will try to have the observability team enable some debug logging on the agent, and see if i can spot something.
Regards Christian Oelsner onsdag den 18. januar 2023 kl. 14.14.23 UTC+1 skrev Brian Candler: > Looks to me like a problem at the receiver end (i.e. the middleware > Elastic agent, or Elasticsearch itself), i.e. that side has stopped > accepting data. > > Try looking at logs of these to determine why they are no longer accepting > data. > > On Wednesday, 18 January 2023 at 11:29:05 UTC [email protected] wrote: > >> Hello guys. >> >> I am scraping some metrics wich ar ethen shipped of to an Elastic agent >> to be ingested into Elasticsearch. All seems fine to start with, but after >> some time, metrics stop comming in, and the prometheus logs shows a lot of >> entries like this: >> >> ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote >> level=warn remote_name=010ca8 url= >> http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed >> to send batch, retrying" err="Post \" >> http://agent-svc.observability.svc.cluster.local:9201/write\": context >> deadline exceeded" >> >> ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote >> level=debug remote_name=010ca8 url= >> http://agent-svc.observability.svc.cluster.local:9201/write msg="Not >> downsharding due to being too far behind" >> >> I am guessing that Prometheus is trying to tell me something, but i just >> dont know what. >> >> Checking the TSDB status on the prom UI it tells me that Number of series >> is 8439 wich does not sound like a lot. >> Any help would be very appreciated. >> >> Best regards >> Christian Oelsner >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4cb58947-e401-4213-bd34-8e00096f3807n%40googlegroups.com.
