Hi Ori, Another more temporary brute-force option, while not officially flink, could be building a modified version of the metrics plugin into flink where you manually manipulate the prefixes yourself. It’s actually pretty easy to build the jar, and to test it you drop the jar into the plugin path. I’ve done something similar where I actually filter our a lot of the prefixes that I don’t want, because too many metric points were being generated from some custom metrics. The config for the filter is loaded from the flink conf, you could possibly implement something similar where you pass the job name in each clusters config:
https://github.com/cslotterback/flink/commit/fd8e1f77a83a3ae1253da53596d22471bb6fe902 and https://github.com/cslotterback/flink/commit/ce3797ea46f3321885c4352ecc36b9385b7ca0ce This isn’t what I’d call ideal, but it gets the job done. I would love a generic flink-approved method of configuring Prometheus metrics. Chris From: Ori Popowski <ori....@gmail.com> Date: Monday, June 22, 2020 at 12:22 PM Cc: user <user@flink.apache.org> Subject: [EXTERNAL] Re: Renaming the metrics Thanks for answering. Unrelated to Flink, but if anyone knows a way to rename the metrics inside Prometheus I'd appreciate if you can share. About the push gateway - I think I'll stick with the pull options because it looks like a better fit to the use case On Mon, Jun 22, 2020 at 4:47 PM Chesnay Schepler <ches...@apache.org<mailto:ches...@apache.org>> wrote: There's currently no way to change this. A related enhancement was proposed on FLINK-17495 that would at least allow you to attach a custom label, but the initial implementation wasn't general enough. On 22/06/2020 15:08, Arvid Heise wrote: Hi Ori, I see that the PrometheusPushGatewayReporter [1] has an option for a job name, maybe you can use that. I'm also including Chesnay who probably has more ideas. [1] https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html#prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter<https://urldefense.com/v3/__https:/ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html*prometheuspushgateway-orgapacheflinkmetricsprometheusprometheuspushgatewayreporter__;Iw!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNL143GF4$> On Mon, Jun 22, 2020 at 9:01 AM Ori Popowski <ori....@gmail.com<mailto:ori....@gmail.com>> wrote: I have two Flink clusters sending metrics via Prometheus and they share all the metric names (i.e. flink_taskmanager_job_task_operator_currentOutputWatermark). I want to change the flink_ prefix to something else to distinguish between the clusters (maybe the job-name). How can I do it? Thanks. -- Arvid Heise | Senior Java Developer [https://lh5.googleusercontent.com/ODbO0aq1IqKMfuoy_pw2YH8r6dqDRTq37rg3ytg11FCGJx12jJ1ff_SANPBxTHzSJTUQY9JLuoXq4NB7Om7j6Vq1lg6jIOKz8S5g2VKDGwicbj5fbY09PVb6mD5TdRuWEUvEMZTG]<https://urldefense.com/v3/__https:/www.ververica.com/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNOG4n0F8$> Follow us @VervericaData -- Join Flink Forward<https://urldefense.com/v3/__https:/flink-forward.org/__;!!CQl3mcHX2A!TVuthJHVtx7kmK6h8pk4bAuSzLu0Gk9Tmxm6JKPVsoOxtzG44_wJ9viTv4VFM2PNm4cAkLw$> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
