Thanks Brian. This is helpful.
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: [email protected] <[email protected]> on
behalf of Brian Candler <[email protected]>
Sent: Sunday, January 29, 2023 6:46:12 PM
To: Prometheus Users <[email protected]>
Subject: Re: [prometheus-users] Re: Prometheus same host multiple endpoints
Probably the ideal would be to fix the problem at source: have an alerting rule
which tells you if the same JVM is exposing both sets of metrics, and push it
back to the JVM owner.
Otherwise your dashboard metrics will need to take into account the duplicates.
You could simplify this a bit by having recording rules which take one or the
other, taking care that some labels will match and some won't. e.g.
expr: foo{job="jmx"} on ignoring(job) foo{job="springboot")
Then you can remote write this timeseries.
On Saturday, 28 January 2023 at 15:07:55 UTC [email protected] wrote:
Hi Brian/Stuart:
The prometheus implementation has around 2000+JVMs and uses two approaches (JMX
exporter or SpringBoot Actuator/Micrometer). The implementation is up to the
JVM owner.
JMX exporter exposes metrics at http://host:port/metrics whereas Micrometer
exposes them at http://host:port/actuator/metrics. Given we don't know where
metrics might be exposed, we have both of them mentioned in prometheus.yaml.
This works fine, given we expect metrics to only come from one source. However,
we have found cases where they can come through both http://host:port/metrics
and http://host:port/actuator/metrics. For these cases, there are duplicate
metrics stored (it is redundant data) and also gets double counted in
dashboards.
It might be quite possible there are incorrect implementations where JVM is
configured to use both JMX exporter and Actuator/Micrometer. Thinking of a way
of how we can stop storing the duplicates as we find and correct the
implementations.
The targets to be scrapped are stored in json files which are referred to by
prometheus.yaml.
We will be able to help with examples of json files and prometheus.yaml if
required. Let us know.
Thanks,
Naveen
On Mon, Jan 23, 2023 at 9:08 PM Brian Candler <[email protected]> wrote:
Can you give an specific example of "same metrics are published on two
different endpoints" ?
You might mean:
- two different metric names
- the same metric name, but different labels
And it might be that you're scraping the same target twice, or you're scraping
one target but that target is (for some reason) returning duplicates in the
scrape results. Or you might have a more complex scenario, e.g. multiple
prometheus servers scraping for redundancy, and then you're combining the
results together somehow.
> Is it possible to pick one endpoint and discard the other while writing a
> PromQL query ?
Sure. Just filter in the PromQL query. For example, if you have
foo{aaa="bbb",ccc="ddd"} 123.0
foo{aaa="bbb",ccc="fff"} 123.0
and you consider the one with ccc="fff" to be a "duplicate" metric, then
foo{ccc!="fff"}
might be what you want.
Otherwise, you can avoid ingesting the duplicate metrics:
- by not scraping the second set in the first place
- if they all come from the same scrape, then using metric_relabel_configs to
drop the metrics that you don't want to keep
On Monday, 23 January 2023 at 14:40:40 UTC [email protected] wrote:
Hi,
We have a situation where same metrics are published on two different
endpoints. Is it possible to pick one endpoint and discard the other while
writing a PromQL query ?
Is it possible to configure Prometheus to collect metrics from only one
endpoint?
/ Kishore
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com<https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com<https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/PU4P216MB1088E9BF3F4D64C819547D20A6D39%40PU4P216MB1088.KORP216.PROD.OUTLOOK.COM.
