[
https://issues.apache.org/jira/browse/SOLR-17587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Biscocho updated SOLR-17587:
------------------------------------
Description:
Solr's Prometheus writer duplicates `# TYPE <metric name> <prometheus metric
type>` in it's exposition format for core registry metrics.
For example this appears twice in it's output:
{code:java}
# TYPE solr_metrics_core_average_request_time gauge
solr_metrics_core_average_request_time{category="ADMIN",collection="foo",core="core_foo_shard9_replica_t351",handler="/admin/file",replica="replica_t351",shard="shard9"}
0.0
...
# TYPE solr_metrics_core_average_request_time gauge{code}
This is technically not allowed per [Prometheus Exposition
format|https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#:~:text=Only%20one%20TYPE%20line%20may%20exist%20for%20a%20given%20metric%20name.]
This happens because each Dropwizard registry is per core, but for Prometheus
compatible exposition format upon exporting, it needs to be 1 registry for all
cores on a single host, otherwise there will be duplicate `TYPE` formats even
though all metrics are unique for its tags/attributes.
Funnily enough, prometheus upstream collector does not do this verification and
accepts the metrics anyways just fine Solr -> Prometheus -> Grafana.
But depending on the technologies prometheus exposition verification, this will
fail. For example
[Telegraf|https://github.com/influxdata/telegraf/blob/master/plugins/inputs/prometheus/README.md]:
{code:java}
-12-09T16:56:01Z E! [inputs.prometheus] Error in plugin: error reading metrics
for "http://127.0.0.1:8983/solr/admin/metrics?wt=prometheus": decoding response
failed: text format parsing error in line 568: second TYPE line for metric name
"solr_metrics_core_average_request_time", or TYPE reported after samples {code}
This shouldn't be a blocker if you are pushing metrics to prometheus collector
directly.
was:
Solr's Prometheus writer duplicates `# TYPE <metric name> <prometheus metric
type>` in it's exposition format for core registry metrics.
For example this appears twice in it's output:
{code:java}
# TYPE solr_metrics_core_average_request_time gauge
solr_metrics_core_average_request_time{category="ADMIN",collection="foo",core="core_foo_shard9_replica_t351",handler="/admin/file",replica="replica_t351",shard="shard9"}
0.0
...
# TYPE solr_metrics_core_average_request_time gauge{code}
This is technically not allowed per [Prometheus Exposition
format|https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#:~:text=Only%20one%20TYPE%20line%20may%20exist%20for%20a%20given%20metric%20name.]
This happens because each Dropwizard registry is per core, but for Prometheus
compatible exposition format upon exporting, it needs to be 1 registry for all
cores on a single host, otherwise there will be duplicate `TYPE` formats even
though all metrics are unique for its tags/attributes.
Funnily enough, prometheus upstream collector does not do this verification and
accepts the metrics anyways just fine Solr -> Prometheus -> Grafana.
But depending on the technologies prometheus exposition verification, this will
fail. For example
[Telegraf|https://github.com/influxdata/telegraf/blob/master/plugins/inputs/prometheus/README.md]:
{code:java}
-12-09T16:56:01Z E! [inputs.prometheus] Error in plugin: error reading metrics
for "http://127.0.0.1:8983/solr/admin/metrics?wt=prometheus": decoding response
failed: text format parsing error in line 568: second TYPE line for metric name
"solr_metrics_core_average_request_time", or TYPE reported after samples {code}
This shouldn't be a blocker if you are pushing metrics to prometheus collector
directly.
> Prometheus Writer Duplicate TYPE exposition format
> --------------------------------------------------
>
> Key: SOLR-17587
> URL: https://issues.apache.org/jira/browse/SOLR-17587
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 9.7
> Reporter: Matthew Biscocho
> Priority: Minor
>
> Solr's Prometheus writer duplicates `# TYPE <metric name> <prometheus metric
> type>` in it's exposition format for core registry metrics.
> For example this appears twice in it's output:
> {code:java}
> # TYPE solr_metrics_core_average_request_time gauge
> solr_metrics_core_average_request_time{category="ADMIN",collection="foo",core="core_foo_shard9_replica_t351",handler="/admin/file",replica="replica_t351",shard="shard9"}
> 0.0
> ...
> # TYPE solr_metrics_core_average_request_time gauge{code}
> This is technically not allowed per [Prometheus Exposition
> format|https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#:~:text=Only%20one%20TYPE%20line%20may%20exist%20for%20a%20given%20metric%20name.]
> This happens because each Dropwizard registry is per core, but for Prometheus
> compatible exposition format upon exporting, it needs to be 1 registry for
> all cores on a single host, otherwise there will be duplicate `TYPE` formats
> even though all metrics are unique for its tags/attributes.
> Funnily enough, prometheus upstream collector does not do this verification
> and accepts the metrics anyways just fine Solr -> Prometheus -> Grafana.
> But depending on the technologies prometheus exposition verification, this
> will fail. For example
> [Telegraf|https://github.com/influxdata/telegraf/blob/master/plugins/inputs/prometheus/README.md]:
> {code:java}
> -12-09T16:56:01Z E! [inputs.prometheus] Error in plugin: error reading
> metrics for "http://127.0.0.1:8983/solr/admin/metrics?wt=prometheus":
> decoding response failed: text format parsing error in line 568: second TYPE
> line for metric name "solr_metrics_core_average_request_time", or TYPE
> reported after samples {code}
> This shouldn't be a blocker if you are pushing metrics to prometheus
> collector directly.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
