Thanks a lot, Brian. I think I should create an issue. On Monday, May 20, 2024 at 1:37:51 PM UTC+8 Brian Candler wrote:
> > server returned HTTP status 500 Internal Server Error: too old sample > > This is not the server failing to process the data; it's the client > supplying invalid data. You found that this has been fixed to a 400. > > > server returned HTTP status 500 Internal Server Error: label name > \"prometheus\" is not unique: invalid sample > > I can't speak for the authors, but it looks to me like that should be a > 400 as well. > > On Monday 20 May 2024 at 04:52:03 UTC+1 koly li wrote: > >> Sorry for my poor description. Here is the story: >> >> 1) At first, we are using prometheus v2.47 >> >> Then we found all metrics are missing, we check the prometheus log and >> prometheus agent log: >> >> prometheus log(lots of lines): >> ts=2024-04-19T20:33:26.485Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> ts=2024-04-19T20:33:26.539Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> ts=2024-04-19T20:33:26.626Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> ts=2024-04-19T20:33:26.775Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> ts=2024-04-19T20:33:27.042Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> ts=2024-04-19T20:33:27.552Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> .... >> ts=2024-04-22T03:00:03.327Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> ts=2024-04-22T03:00:08.394Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="too old sample" >> >> prometheus agent logs: >> ts=2024-04-19T20:33:26.517Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-19T20:34:29.714Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-19T20:35:30.113Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-19T20:36:30.478Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> .... >> ts=2024-04-22T02:56:57.281Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-22T02:57:57.624Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-22T02:58:57.943Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-22T02:59:58.267Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> ts=2024-04-22T03:00:58.733Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: too old sample" >> >> Then we check the codes: >> >> https://github.com/prometheus/prometheus/blob/release-2.47/storage/remote/write_handler.go#L77 >> >> The "too old sample" is considered an 500. And the agent keeps retrying >> (exit only when the error is not Recoverable, and 500 is considered >> Recoverable): >> >> https://github.com/prometheus/prometheus/blob/release-2.51/storage/remote/queue_manager.go#L1670 >> >> > You may have come across a bug where a *particular* piece of data being >> sent by the agent was causing a *particular* version of prometheus to fail >> with a 5xx internal error every time. The logs should make it clear if this >> was happening. >> We guess there is one or more samples with too old timestamps which cause >> the problem. One or more samples with "too old timestamp" cause the >> prometheus agent to retry forever (agent receives 500) , which prevents new >> samples to be sent. >> >> Because there is no logging about the incorrect samples, we cannot figure >> out what these samples are. It logs samples only for several errors: >> https://github.com/prometheus/prometheus/blob/release-2.47/storage/remote/write_handler.go#L132 >> >> > The fundamental issue here is, why should restarting the *agent* cause >> the prometheus *server* to stop returning 500 errors? >> We restart the agent 1~2 days after the problem occurs. The new data does >> not contain too old samples. That's why 500 errors disappear. >> >> 2) then we upgrade to v2.51 >> The new version returns 400 for "too old samples": >> >> https://github.com/prometheus/prometheus/blob/release-2.51/storage/remote/write_handler.go#L72 >> >> However, we encountered another 500: >> prometheus agent log: >> ts=2024-05-11T08:42:01.235Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: label name \"prometheus\" is not unique: invalid sample" >> ts=2024-05-11T08:42:02.749Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: label name \"service\" is not unique: invalid sample" >> ts=2024-05-11T08:42:02.798Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: label name \"resourceType\" is not unique: invalid sample" >> ts=2024-05-11T08:42:02.851Z caller=dedupe.go:112 component=remote >> level=warn remote_name=prometheus-k8s-0 url= >> https://prometheus-k8s-0.monitoring:9091/api/v1/write msg="Failed to >> send batch, retrying" err="server returned HTTP status 500 Internal Server >> Error: label name \"namespace\" is not unique: invalid sample" >> >> we modify the code to log samples, then we get the prometheus log: >> ts=2024-05-11T08:42:26.603Z caller=write_handler.go:134 level=error >> component=web msg="unknown error from remote write" err="label name >> \"resourceId\" is not unique: invalid sample" >> series="{__name__=\"ovs_vswitchd_interface_resets_total\", >> clusterName=\"clustertest150\", clusterRegion=\"region0\", >> clusterZone=\"zone1\", container=\"kube-rbac-proxy\", >> endpoint=\"ovs-metrics\", hostname=\"20230428-wangbo-dev16\", >> if_name=\"veth99fa6555\", instance=\"10.253.58.238:9983\", >> job=\"net-monitor-vnet-ovs\", namespace=\"net-monitor\", >> pod=\"net-monitor-vnet-ovs-66bdz\", prometheus=\"monitoring/agent-0\", >> prometheus_replica=\"prometheus-agent-0-0\", >> resourceId=\"port-naqoi5tmkg5lrt0ubw\", resourceId=\"blb-74se39mqa9k3\", >> resourceType=\"Port\", resourceType=\"BLB\", rs_ip=\"10.0.0.3\", >> service=\"net-monitor-vnet-ovs\", service=\"net-monitor-vnet-ovs\", >> subnet_Id=\"snet-ztojflwrnd08xf5idw\", vip=\"11.4.2.64\", >> vpc_Id=\"vpc-6ss1uz29ctpfv0eqbj\", vpcid=\"11.4.2.64\"}" >> timestamp=1715349156000 >> ts=2024-05-11T08:42:26.603Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="label name >> \"resourceId\" is not unique: invalid sample" >> ts=2024-05-11T08:42:26.967Z caller=write_handler.go:134 level=error >> component=web msg="unknown error from remote write" err="label name >> \"service\" is not unique: invalid sample" >> series="{__name__=\"rest_client_request_size_bytes_bucket\", >> clusterName=\"clustertest150\", clusterRegion=\"region0\", >> clusterZone=\"zone1\", container=\"kube-scheduler\", endpoint=\"https\", >> host=\"127.0.0.1:6443\", instance=\"10.253.58.236:10259\", >> job=\"scheduler\", le=\"262144\", namespace=\"kube-scheduler\", >> pod=\"kube-scheduler-20230428-wangbo-dev14\", >> prometheus=\"monitoring/agent-0\", >> prometheus_replica=\"prometheus-agent-0-0\", resourceType=\"NETWORK-HOST\", >> service=\"scheduler\", service=\"net-monitor-vnet-ovs\", verb=\"POST\"}" >> timestamp=1715349164522 >> ts=2024-05-11T08:42:26.967Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="label name >> \"service\" is not unique: invalid sample" >> ts=2024-05-11T08:42:27.091Z caller=write_handler.go:134 level=error >> component=web msg="unknown error from remote write" err="label name >> \"prometheus_replica\" is not unique: invalid sample" >> series="{__name__=\"workqueue_work_duration_seconds_sum\", >> clusterName=\"clustertest150\", clusterRegion=\"region0\", >> clusterZone=\"zone1\", endpoint=\"https\", instance=\"21.100.10.52:8443\", >> job=\"metrics\", name=\"ResourceSyncController\", >> namespace=\"service-ca-operator\", >> pod=\"service-ca-operator-645cfdbfb6-rjr4z\", >> prometheus=\"monitoring/agent-0\", >> prometheus_replica=\"prometheus-agent-0-0\", >> prometheus_replica=\"prometheus-agent-0-0\", service=\"metrics\"}" >> timestamp=1715349271085 >> ts=2024-05-11T08:42:27.091Z caller=write_handler.go:76 level=error >> component=web msg="Error appending remote write" err="label name >> \"prometheus_replica\" is not unique: invalid sample" >> >> Currently we dont' know why there are duplicated labels. But when the >> server encounters duplicated labels, it returns 500. Then the agent keeps >> retrying, which means new samples cannot be handled. >> >> we set external_labels in prometheus-agent configs: >> global: >> evaluation_interval: 30s >> scrape_interval: 5m >> scrape_timeout: 1m >> external_labels: >> clusterName: clustertest150 >> clusterRegion: region0 >> clusterZone: zone1 >> prometheus: ccos-monitoring/agent-0 >> prometheus_replica: prometheus-agent-0-0 >> keep_dropped_targets: 1 >> >> and the remote write config: >> remote_write: >> - url: https://prometheus-k8s-0.monitoring:9091/api/v1/write >> remote_timeout: 30s >> name: prometheus-k8s-0 >> write_relabel_configs: >> - target_label: __tmp_cluster_id__ >> replacement: 713c30cb-81c3-411d-b4dc-0c775a0f9564 >> action: replace >> - regex: __tmp_cluster_id__ >> action: labeldrop >> bearer_token: XDFSDF... >> tls_config: >> insecure_skip_verify: true >> queue_config: >> capacity: 10000 >> min_shards: 1 >> max_shards: 500 >> max_samples_per_send: 2000 >> batch_send_deadline: 10s >> min_backoff: 30ms >> max_backoff: 5s >> sample_age_limit: 5m >> >> > You are saying that you would prefer the agent to throw away data, >> rather than hold onto the data and try again later when it may succeed. In >> this situation, retrying is normally the correct thing to do. >> Yes, retry is the normal solution. But there should be maximum number of >> retries. We notice that prometheus agent sets the retry nubmers to the >> request header, but it seems the request header is not used by the server. >> >> prometheus-agent sets the retry numbers to request header: >> >> https://github.com/prometheus/prometheus/blob/release-2.51/storage/remote/client.go#L214 >> >> Besides, if some samples is incorrect and others are correct in the same >> request, why don't prometheus server save the correct part and drop the >> wrong part? It is more complicated as retry should be considered, but is it >> possible to save partial data and return 206 when the maximum number of >> retries is reached? >> >> And should prometheus server log samples for all kinds of error? >> >> https://github.com/prometheus/prometheus/blob/release-2.51/storage/remote/write_handler.go#L133 >> >> On Friday, May 17, 2024 at 8:15:04 PM UTC+8 Brian Candler wrote: >> >>> It's difficult to make sense of what you're saying. Without seeing logs >>> from both the agent and the server while this problem was occurring (e.g. >>> `journalctl -eu prometheus`), it's hard to know what was really happening. >>> Also you need to say what exact versions of prometheus and the agent were >>> running. >>> >>> The fundamental issue here is, why should restarting the *agent* cause >>> the prometheus *server* to stop returning 500 errors? >>> >>> >>> > So my question is why 5xx from the promtheus server is considered >>> Recoverable? >>> >>> It is by definition of the HTTP protocol: >>> https://datatracker.ietf.org/doc/html/rfc2616#section-10.5 >>> >>> Actually it depends on exactly which 5xx error code you're talking >>> about, but common 500 and 503 errors are generally transient, meaning there >>> was a problem at the server and the request may succeed if tried again >>> later. If the prometheus server wanted to tell the client that the request >>> was invalid and could never possibly succeed, then it would return a 4xx >>> error. >>> >>> > And I believe there should be a way to exit the loop, for example a >>> maximum times to retry. >>> >>> You are saying that you would prefer the agent to throw away data, >>> rather than hold onto the data and try again later when it may succeed. In >>> this situation, retrying is normally the correct thing to do. >>> >>> You may have come across a bug where a *particular* piece of data being >>> sent by the agent was causing a *particular* version of prometheus to fail >>> with a 5xx internal error every time. The logs should make it clear if this >>> was happening. >>> >>> On Friday 17 May 2024 at 10:02:49 UTC+1 koly li wrote: >>> >>>> Hello all, >>>> >>>> Recently we found that our samples are all lost. After some >>>> investigation, we found: >>>> 1, we are using prometheus agent to send all data to prometheus server >>>> by remote write >>>> 2, the agent sample sending code is in storage\remote\queue_manager.go, >>>> the function is sendWriteRequestWithBackoff() >>>> 3, inside the function, if attempt(the function where request is made >>>> to prometheus server) function returns an Recoverable Error, then it will >>>> retry sending the request >>>> 4, when a Recoverable error is returned? one scenario is the prometheus >>>> server returned 5xx error >>>> 5, I think not every 5xx error is recoverable, and there is no other >>>> way to exit the for loop in sendWriteRequestWithBackoff(). The agent keeps >>>> retrying but every time it receives an 5xx from the server. so we lost all >>>> samples for hours until we restart the agent >>>> >>>> So my question is why 5xx from the promtheus server is considered >>>> Recoverable? And I believe there should be a way to exit the loop, for >>>> example a maximum times to retry. >>>> >>>> It seems that the agent mode is not mature enough to work in production. >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5661de1d-21c5-486c-9177-fa346ebdc922n%40googlegroups.com.
