Matthew Biscocho created SOLR-17587:
---------------------------------------
Summary: Prometheus Writer Duplicate TYPE exposition format
Key: SOLR-17587
URL: https://issues.apache.org/jira/browse/SOLR-17587
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 9.7
Reporter: Matthew Biscocho
Solr's Prometheus writer duplicates `# TYPE <metric name> <prometheus metric
type>` in it's exposition format for core registry metrics.
For example this appears twice in it's output:
{code:java}
# TYPE solr_metrics_core_average_request_time gauge
solr_metrics_core_average_request_time{category="ADMIN",collection="foo",core="core_foo_shard9_replica_t351",handler="/admin/file",replica="replica_t351",shard="shard9"}
0.0
...
# TYPE solr_metrics_core_average_request_time gauge{code}
This is technically not allowed per [Prometheus Exposition
format|https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#:~:text=Only%20one%20TYPE%20line%20may%20exist%20for%20a%20given%20metric%20name.]
This happens because each Dropwizard registry is per core, but for Prometheus
compatible exposition format upon exporting, it needs to be 1 registry for all
cores on a single host, otherwise there will be duplicate `TYPE` formats even
though all metrics are unique for its tags/attributes.
Funnily enough, prometheus upstream collector does not do this verification and
accepts the metrics anyways just fine Solr -> Prometheus -> Grafana.
But depending on the technologies prometheus exposition verification, this will
fail. For example
[Telegraf|https://github.com/influxdata/telegraf/blob/master/plugins/inputs/prometheus/README.md]:
{code:java}
-12-09T16:56:01Z E! [inputs.prometheus] Error in plugin: error reading metrics
for "http://127.0.0.1:8983/solr/admin/metrics?wt=prometheus": decoding response
failed: text format parsing error in line 568: second TYPE line for metric name
"solr_metrics_core_average_request_time", or TYPE reported after samples {code}
This shouldn't be a blocker if you are pushing metrics to prometheus collector
directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
