perosb opened a new issue, #705: URL: https://github.com/apache/solr-operator/issues/705
Solr 8.11 seem to be producing duplicate metrics which is flooding the logs and triggering alerts. This is only when using latest prometheus v2.52.0. Related https://github.com/prometheus/prometheus/issues/14089 This doesn't seem to happen with Solr 9. It still happens when running solr 8.11 and solr-exporter image tag 9.x This is the debug log with 1 collection and the related duplicates: ``` {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_requests_total{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_mean_rate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_mean_rate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_5minRate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_1minRate{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p75_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p75_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_median_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_median_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_requests_total{category=\"QUERY\",handler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_requests_total{category=\"ADMIN\",handler=\"/admin/ping\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.909Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p99_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p99_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p95_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_query_p95_ms{category=\"QUERY\",searchHandler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_requests_total{category=\"ADMIN\",handler=\"/admin/info\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_requests_total{category=\"ADMIN\",handler=\"/admin/metrics\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_time_seconds_total{category=\"ADMIN\",handler=\"/admin/info\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_node_time_seconds_total{category=\"ADMIN\",handler=\"/admin/metrics\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_time_seconds_total{category=\"QUERY\",handler=\"/select\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1777","component":"scrape manager","level":"debug","msg":"Duplicate sample for timestamp","scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","series":"solr_metrics_core_time_seconds_total{category=\"ADMIN\",handler=\"/admin/ping\",internal=\"false\",core=\"kermit_shard1_replica_n1\",collection=\"kermit\",shard=\"shard1\",replica=\"replica_n1\",base_url=\"http://frog-solrcloud-1.frog-solrcloud-headless.solr-test:8983/solr\",cluster_id=\"f7122eee5d\"}","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.910Z"} {"caller":"scrape.go:1738","component":"scrape manager","level":"warn","msg":"Error on ingesting samples with different value but same timestamp","num_dropped":21,"scrape_pool":"serviceMonitor/solr-test/solr-metrics/0","target":"http://172.23.50.53:8080/metrics","ts":"2024-05-15T08:32:26.911Z"} ``` *Disclaimer: this probably is a Solr bug rather than operator?* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
