Cheers. BTW I mistyped that, it should have been "or" not "on".
expr: foo{job="jmx"} *or* ignoring(job) foo{job="springboot"}
On Monday, 30 January 2023 at 02:07:09 UTC [email protected] wrote:
> Thanks Brian. This is helpful.
>
> Sent from Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
> *From:* [email protected] <[email protected]> on
> behalf of Brian Candler <[email protected]>
> *Sent:* Sunday, January 29, 2023 6:46:12 PM
> *To:* Prometheus Users <[email protected]>
> *Subject:* Re: [prometheus-users] Re: Prometheus same host multiple
> endpoints
>
> Probably the ideal would be to fix the problem at source: have an alerting
> rule which tells you if the same JVM is exposing both sets of metrics, and
> push it back to the JVM owner.
>
> Otherwise your dashboard metrics will need to take into account the
> duplicates.
>
> You could simplify this a bit by having recording rules which take one or
> the other, taking care that some labels will match and some won't. e.g.
>
> expr: foo{job="jmx"} on ignoring(job) foo{job="springboot")
>
> Then you can remote write this timeseries.
>
> On Saturday, 28 January 2023 at 15:07:55 UTC [email protected] wrote:
>
> Hi Brian/Stuart:
>
> The prometheus implementation has around 2000+JVMs and uses two approaches
> (JMX exporter or SpringBoot Actuator/Micrometer). The implementation is up
> to the JVM owner.
>
> JMX exporter exposes metrics at http://host:port/metrics whereas
> Micrometer exposes them at http://host:port/actuator/metrics. Given we
> don't know where metrics might be exposed, we have both of them mentioned
> in prometheus.yaml.
>
> This works fine, given we expect metrics to only come from one source.
> However, we have found cases where they can come through both
> http://host:port/metrics and http://host:port/actuator/metrics. For these
> cases, there are duplicate metrics stored (it is redundant data) and also
> gets double counted in dashboards.
>
> It might be quite possible there are incorrect implementations where JVM
> is configured to use both JMX exporter and Actuator/Micrometer. Thinking of
> a way of how we can stop storing the duplicates as we find and correct the
> implementations.
>
> The targets to be scrapped are stored in json files which are referred
> to by prometheus.yaml.
>
> We will be able to help with examples of json files and prometheus.yaml if
> required. Let us know.
>
> Thanks,
> Naveen
>
>
>
> On Mon, Jan 23, 2023 at 9:08 PM Brian Candler <[email protected]> wrote:
>
> Can you give an specific example of "same metrics are published on two
> different endpoints" ?
>
> You might mean:
> - two different metric names
> - the same metric name, but different labels
>
> And it might be that you're scraping the same target twice, or you're
> scraping one target but that target is (for some reason) returning
> duplicates in the scrape results. Or you might have a more complex
> scenario, e.g. multiple prometheus servers scraping for redundancy, and
> then you're combining the results together somehow.
>
> > Is it possible to pick one endpoint and discard the other while writing
> a PromQL query ?
>
> Sure. Just filter in the PromQL query. For example, if you have
>
> foo{aaa="bbb",ccc="ddd"} 123.0
> foo{aaa="bbb",ccc="fff"} 123.0
>
> and you consider the one with ccc="fff" to be a "duplicate" metric, then
>
> foo{ccc!="fff"}
>
> might be what you want.
>
> Otherwise, you can avoid ingesting the duplicate metrics:
> - by not scraping the second set in the first place
> - if they all come from the same scrape, then using metric_relabel_configs
> to drop the metrics that you don't want to keep
>
> On Monday, 23 January 2023 at 14:40:40 UTC [email protected] wrote:
>
> Hi,
> We have a situation where same metrics are published on two different
> endpoints. Is it possible to pick one endpoint and discard the other while
> writing a PromQL query ?
> Is it possible to configure Prometheus to collect metrics from only one
> endpoint?
>
> / Kishore
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com
>
> <https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/861c7610-45ee-4fe3-981e-ce9de36fccddn%40googlegroups.com.
