Thanks for the suggestion.
After digging a bit, we've found it most convenient to just add labels to
all our Prometheus queries, like this:
flink_taskmanager_job_task_operator_currentOutputWatermark{job_name="<job-name>"}
The job_name label will be exposed if you run your job with a job name like
this:
senv.execute("<job-name>")
On Mon, Jun 22, 2020 at 8:01 PM Slotterback, Chris <
chris_slotterb...@comcast.com> wrote:
> Hi Ori,
>
>
>
> Another more temporary brute-force option, while not officially flink,
> could be building a modified version of the metrics plugin into flink where
> you manually manipulate the prefixes yourself. It’s actually pretty easy to
> build the jar, and to test it you drop the jar into the plugin path. I’ve
> done something similar where I actually filter our a lot of the prefixes
> that I don’t want, because too many metric points were being generated from
> some custom metrics. The config for the filter is loaded from the flink
> conf, you could possibly implement something similar where you pass the job
> name in each clusters config:
>
>
>
>
> https://github.com/cslotterback/flink/commit/fd8e1f77a83a3ae1253da53596d22471bb6fe902
>
> and
>
>
> https://github.com/cslotterback/flink/commit/ce3797ea46f3321885c4352ecc36b9385b7ca0ce
>
>
>
> This isn’t what I’d call ideal, but it gets the job done. I would love a
> generic flink-approved method of configuring Prometheus metrics.
>
>
>
> Chris
>
>
>
>
>
> *From: *Ori Popowski <ori....@gmail.com>
> *Date: *Monday, June 22, 2020 at 12:22 PM
> *Cc: *user <user@flink.apache.org>
> *Subject: *[EXTERNAL] Re: Renaming the metrics
>
>
>
> Thanks for answering.
>
>
>
> Unrelated to Flink, but if anyone knows a way to rename the metrics inside
> Prometheus I'd appreciate if you can share.
>
>
>
> About the push gateway - I think I'll stick with the pull options because
> it looks like a better fit to the use case
>
>
>
> On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <ches...@apache.org>
> wrote:
>
> There's currently no way to change this.
>
>
>
> A related enhancement was proposed on FLINK-17495 that would at least
> allow you to attach a custom label, but the initial implementation wasn't
> general enough.
>
>
>
> On 22/06/2020 15:08, Arvid Heise wrote:
>
> Hi Ori,
>
>
>
> I see that the PrometheusPushGatewayReporter [1] has an option for a job
> name, maybe you can use that.
>
>
>
> I'm also including Chesnay who probably has more ideas.
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter
> <https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html*prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter__;Iw!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNL143GF4$>
>
>
>
> On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <ori....@gmail.com> wrote:
>
> I have two Flink clusters sending metrics via Prometheus and they share
> all the metric names (i.e.
> flink_taskmanager_job_task_operator_currentOutputWatermark).
>
>
>
> I want to change the flink_ prefix to something else to distinguish
> between the clusters (maybe the job-name).
>
>
>
> How can I do it?
>
>
>
> Thanks.
>
>
>
> --
>
> *Arvid Heise *| Senior Java Developer
>
>
> <https://urldefense.com/v3/__https:/www.ververica.com/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNOG4n0F8$>
>
>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward
> <https://urldefense.com/v3/__https:/flink-forward.org/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNm4cAkLw$>
> - The Apache Flink Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
>
>
>
