Flink metrics to Prometheus on Kubernetes

Hi, I have a few Flink jobs running on Kubernetes using the Flink Kubernetes Operator. By following the documentation [1] I was able to set up monitoring for the Operator itself. As for the jobs themselves, I'm a bit confused about how to properly set it up. Here's my FlinkDeployment configuration

Re: Question about Flink metrics

Hi Neha, For the jobs you care about, you can attach additional labels using `scope-variables-additional` [1]. The example located in the same page showcases how you can configure KV pairs in its map configuration. Be sure to replace the reporter name with the name of your prometheus reporter! [1

Question about Flink metrics

Hello, I have a question about the Prometheus metrics. I am able to fetch the metrics from the following expression. sum(flink_jobmanager_job_numRestarts{job_name="$job_name"}) by (job_name) Now I am interested in only a few jobs and I want to give them a label. How to achieve this? How to give an

Removing labels from Flink metrics

Hi Team, Is it possible to remove a few labels from Flink operator scope metrics as we are noticing that sometimes those labels are too large and hence causing unnecessary load at our monitoring platform. One such label is operator_name. Thanks and Regards , Surendra Lalwani -- IMPORTANT NOTIC

Flink metrics flattened after Job restart

Hi Flink Community We are using Flink version 1.13.5 for our application and every time the job restarts, Flink Job metrics are flattened following the restart. For e.g. we are using lastCheckpointDuration and on 05/05 our job restarted and at the same time the checkpoint duration metric flattened

Re: Flink metrics via permethous or opentelemerty

-monitoring.html On Wed, Feb 23, 2022 at 8:42 AM Sigalit Eliazov wrote: > Hello. I am looking for a way to expose flink metrics via opentelemerty to > the gcp could monitoring dashboard. > Does anyone has experience with that? > > If it is not directly possible we thought about using

Flink metrics via permethous or opentelemerty

Hello. I am looking for a way to expose flink metrics via opentelemerty to the gcp could monitoring dashboard. Does anyone has experience with that? If it is not directly possible we thought about using permethous as a middlewere. If you have experience with that i would appreciate any guidance

Re: regarding flink metrics

Your best bet is to create a custom reporter that does this calculation. You could either wrap the reporter, subclass is, or fork it. In any case, https://github.com/apache/flink/tree/master/flink-metrics/flink-metrics-datadog should be a good starting point. On 01/02/2022 13:26, Jessy Ping

regarding flink metrics

Hi Team, We are using datadog and its http reporter( packaged in flink image) for sending metrics from flink application. We do have a requirement for setting tags with values calculated at runtime for the custom metrics emitted from Flink. Currently, it is impossible to assign tags at runtime. Is

Re: Flink Metrics Naming

Some more background on MetricGroups: Internally there (mostly) 3 types of metric groups: On the one hand we have the ComponentMetricGroups (like TaskManagerMetricGroup) that describe a high-level Flink entity, which just add a constant expression to the logical scope(like taskmanager, task etc

Re: Flink Metrics Naming

Upon further inspection, it seems like the user scope is not universal (i.e. comes through the connectors and not UDFs (like rich map function)), but the question still stands if the process makes sense. > On Jun 1, 2021, at 10:38 AM, Mason Chen wrote: > > Makes sense. We are primarily concern

Re: Flink Metrics Naming

Makes sense. We are primarily concerned with removing the metric labels from the names as the user metrics get too long. i.e. the groups from `addGroup` are concatenated in the metric name. Do you think there would be any issues with removing the group information in the metric name and putting

Re: Flink Metrics Naming

The uniqueness of metrics and the naming of the Prometheus reporter are somewhat related but also somewhat orthogonal. Prometheus works similar to JMX in that the metric name (e.g., taskmanager.job.task.operator.numRecordsIn) is more or less a _class_ of metrics, with tags/labels allowing you

Re: Flink Metrics Naming

Hi Mason, The idea is that a metric is not uniquely identified by its name alone but instead by its path. The groups in which it is defined specify this path (similar to directories). That's why it is valid to specify two metrics with the same name if they reside in different groups. I think Prom

Flink Metrics Naming

Can anyone give insight as to why Flink allows 2 metrics with the same “name”? For example, getRuntimeContext.addGroup(“group”, “group1”).counter(“myMetricName”); And getRuntimeContext.addGroup(“other_group”, “other_group1”).counter(“myMetricName”); Are totally valid. It seems that it has l

Re: Flink Metrics emitted from a Kubernetes Application Cluster

This is currently not possible. See also FLINK-8358 On 4/9/2021 4:47 AM, Claude M wrote: Hello, I've setup Flink as an Application Cluster in Kubernetes. Now I'm looking into monitoring the Flink cluster in Datadog. This is what is configured in the flink-conf.yaml to emit metrics: metrics.

Flink Metrics emitted from a Kubernetes Application Cluster

Hello, I've setup Flink as an Application Cluster in Kubernetes. Now I'm looking into monitoring the Flink cluster in Datadog. This is what is configured in the flink-conf.yaml to emit metrics: metrics.scope.jm: flink.jobmanager metrics.scope.jm.job: flink.jobmanager.job metrics.scope.tm: flink

Re: Flink Metrics

Hi, 1) Do you want to output those metrics as Flink metrics? Or output those "metrics"/counters as values to some external system (like Kafka)? The problem discussed in [1], was that the metrics (Counters) were not fitting in memory, so David suggested to hold them on Flink's stat

Flink Metrics

Hi flinksters, Scenario: We have cdc messages from our rdbms(various tables) flowing to Kafka. Our flink job reads the CDC messages and creates events based on certain rules. I am using Prometheus and grafana. Following are there metrics that i need to calculate A) Number of CDC messages wrt

Re: Tag flink metrics to job name

hmm...in a roundabout way this could be possible I suppose. For a given job, search through your metrics for some job metric (like numRestarts on the JM, or any task metric for TMs), and from that you should be able to infer the JM/TM that belongs to that (based on the TM ID / host information

Re: Tag flink metrics to job name

Is there a way I can look into say for a specific job what’s the cpu usage or memory usage of the yarn containers when multiple jobs are running on the same cluster. Also, the issue am trying to resolve is I’m seeing high memory usage for one of the containers I want isolate the issue with one job

Re: Tag flink metrics to job name

No, Job-/TaskManager metrics cannot be tagged with the job name. The reason is that this only makes sense for application clusters (opposed to session clusters), but we don't differentiate between the two when it comes to metrics. On 2/19/2021 3:59 AM, bat man wrote: I meant the Flink jobname

Re: Tag flink metrics to job name

I meant the Flink jobname. I’m using the below reporter - metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter Is there any way to tag job names to the task and job manager metrics. Thanks, Hemant On Fri, 19 Feb 2021 at 12:40 AM, Chesnay Schepler wrote: > When

Re: Tag flink metrics to job name

When you mean "job_name", are you referring to the Prometheus concept of jobs, of the one of Flink? Which of Flink prometheus reporters are you using? On 2/17/2021 7:37 PM, bat man wrote: Hello there, I am using prometheus to push metrics to prometheus and then use grafana for visualization.

Tag flink metrics to job name

Hello there, I am using prometheus to push metrics to prometheus and then use grafana for visualization. There are metrics like - flink_taskmanager_Status_JVM_CPU_Load, flink_taskmanager_Status_JVM_CPU_Load, flink_taskmanager_Status_JVM_CPU_Time etc which do not gives job_name. It is tied to an in

Re: Default Flink Metrics Graphite

orChain.java:353) >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:144) >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433) >>>>&

Re: Default Flink Metrics Graphite

che.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461) >>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) >>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >>>> at java.lang.Thread.run(Thread.java:

Re: Default Flink Metrics Graphite

gt;> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >>> at java.lang.Thread.run(Thread.java:748) >>> Regards, >>> Vijay >>> >>> >>> On Wed, Aug 26, 2020 at 7:53 AM Chesnay Schepler >>> wrote: >>> >

Re: Default Flink Metrics Graphite

Schepler >> wrote: >> >>> metrics.reporter.grph.class: >>> org.apache.flink.metrics.graphite.GraphiteReporter >>> >>> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphite

Re: Default Flink Metrics Graphite

>> >> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitereporter >> >> On 26/08/2020 16:40, Vijayendra Yadav wrote: >> >> Hi Dawid, >> >> I have 1.10.0 version of flink. W

Re: Default Flink Metrics Graphite

ion of flink. What is alternative for this version ? > > Regards, > Vijay > > > On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz > wrote: > >  > > Hi Vijay, > > I think the problem might be that you are using a wrong version of the > reporter. > > Y

Re: Default Flink Metrics Graphite

version of flink. What is alternative for this version ? Regards, Vijay On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz wrote:  Hi Vijay, I think the problem might be that you are using a wrong version of the reporter. You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a

Re: Default Flink Metrics Graphite

erver.md> >>> 81268126statsd admin >>> <https://github.com/etsy/statsd/blob/master/docs/admin_interface.md> >>> >>> 2) WebUI:  >>> >>> >>> >>> >>> >>> 3) Run Flink example Job. >>> ./bin/f

Re: Default Flink Metrics Graphite

reaming_2.11-1.11-SNAPSHOT-SocketWindowWordCount.jar >> --port >> >> with conf/flink-conf.yaml set as: >> >> metrics.reporter.grph.factory.class: >> org.apache.flink.metrics.graphite.GraphiteReporterFactory >> metrics.reporter.grph.host: localhost >>

Re: Default Flink Metrics Graphite

Hi Vijay, I think the problem might be that you are using a wrong version of the reporter. You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a plugin, but it was migrated to plugins in 1.11 only[1]. I'd recommend trying it out with the same 1.11 version of Flink and Gra

Re: Default Flink Metrics Graphite

erFactory metrics.reporter.grph.host: localhost metrics.reporter.grph.port: 2003 metrics.reporter.grph.protocol: TCP metrics.reporter.grph.interval: 1 SECONDS and graphite jar: plugins/flink-metrics-graphite/flink-metrics-graphite-1.10.0.jar 4) Can't see any activity in webui graphite. Could you r

Re: Default Flink Metrics Graphite

t; below are correct? >> >> 1) Updated flink-conf.yaml >> >> metrics.reporter.grph.factory.class: >> org.apache.flink.metrics.graphite.GraphiteReporterFactory >> metrics.reporter.grph.host: port >> metrics.reporter.grph.port: 9109 >> metrics.reporter.grph.protocol: TCP >&g

Re: Default Flink Metrics Graphite

> metrics.reporter.grph.interval: 30 SECONDS > > 2) Added Graphite jar in plugin folder : > > ll */usr/lib/flink/plugins/metric/* > *flink-metrics-graphite-1.10.0.jar* > > 3) Looking metrics in graphite server: > > http://port:8080/metrics <http://10.108.58.63:8080/metrics> > > Note: No code change is done. > > Regards, > Vijay > > >

Default Flink Metrics Graphite

class: org.apache.flink.metrics.graphite.GraphiteReporterFactory metrics.reporter.grph.host: port metrics.reporter.grph.port: 9109 metrics.reporter.grph.protocol: TCP metrics.reporter.grph.interval: 30 SECONDS 2) Added Graphite jar in plugin folder : ll */usr/lib/flink/plugins/metric/* *flink-metrics-graphite-1.10.0.jar* 3) Looking metrics in gra

Re: A query on Flink metrics in kubernetes

theus scraping Flink metrics data with application running in kubernetes cluster. If taskmanager is running on multiple nodes, and prometheus requests for the metrics data, then is that request directed to one of the nodes(based on some strategy, like round-robin) or is data aggregated from all the

A query on Flink metrics in kubernetes

Hi, I have a query regarding prometheus scraping Flink metrics data with application running in kubernetes cluster. If taskmanager is running on multiple nodes, and prometheus requests for the metrics data, then is that request directed to one of the nodes(based on some strategy, like round

Re: Logging Flink metrics

try again, and report back. >>>>> >>>>> On 06/07/2020 16:35, Chesnay Schepler wrote: >>>>> >>>>> Please enable debug logging and search for warnings from the metric >>>>> groups/registry/reporter. >>>&

Re: Logging Flink metrics

With regards Manish On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: Have you looked at the SLF4J reporter? https://ci.apache.org/projects/flink/flink-docs-release-

Re: Logging Flink metrics

>>>> to me directly. >>>> >>>> On 06/07/2020 16:29, Manish G wrote: >>>> >>>> Job is an infinite streaming one, so it keeps going. Flink >>>> configuration is as: >>>> >>>> metrics.reporter.slf4j.class: >>>> org.apac

Re: Logging Flink metrics

https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: > Hi, > > Is it possible to log Flink metrics in application logs apart from > publishing it to Prometheus? > > With regards

Re: Logging Flink metrics

gt; On Mon, Jul 6, 2020 at 7:57 PM Chesnay Schepler >>> wrote: >>> >>>> How long did the job run for, and what is the configured interval? >>>> >>>> >>>> On 06/07/2020 15:51, Manish G wrote: >>>> >>>> Hi, >>>> >>>> Thanks for this. >>>> >>>> I did the configuration as mentioned at the link(changes in >>>> flink-conf.yml, copying the jar in lib directory), and registered the Meter >>>> with metrics group and invoked markEvent() method in the target code. But I >>>> don't see any related logs. >>>> I am doing this all on my local computer. >>>> >>>> Anything else I need to do? >>>> >>>> With regards >>>> Manish >>>> >>>> On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler >>>> wrote: >>>> >>>>> Have you looked at the SLF4J reporter? >>>>> >>>>> >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter >>>>> >>>>> On 06/07/2020 13:49, Manish G wrote: >>>>> > Hi, >>>>> > >>>>> > Is it possible to log Flink metrics in application logs apart from >>>>> > publishing it to Prometheus? >>>>> > >>>>> > With regards >>>>> >>>>> >>>>> >>>> >>> >>> >> >

Re: Logging Flink metrics

Manish On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler mailto:ches...@apache.org>> wrote: Have you looked at the SLF4J reporter? https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflin

Re: Logging Flink metrics

th metrics group and invoked markEvent() method in the target code. But I >>> don't see any related logs. >>> I am doing this all on my local computer. >>> >>> Anything else I need to do? >>> >>> With regards >>> Manish >>> >>> On Mon, Jul 6, 2020 at 5:24 PM Chesnay Schepler >>> wrote: >>> >>>> Have you looked at the SLF4J reporter? >>>> >>>> >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter >>>> >>>> On 06/07/2020 13:49, Manish G wrote: >>>> > Hi, >>>> > >>>> > Is it possible to log Flink metrics in application logs apart from >>>> > publishing it to Prometheus? >>>> > >>>> > With regards >>>> >>>> >>>> >>> >> >> >

Re: Logging Flink metrics

rter? https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: > Hi, > > Is it possible to log Flink

Re: Logging Flink metrics

6, 2020 at 5:24 PM Chesnay Schepler >> wrote: >> >>> Have you looked at the SLF4J reporter? >>> >>> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter >>> >>> On 06/07/2020 13:49, Manish G wrote: >>> > Hi, >>> > >>> > Is it possible to log Flink metrics in application logs apart from >>> > publishing it to Prometheus? >>> > >>> > With regards >>> >>> >>> >> > >

Re: Logging Flink metrics

On 06/07/2020 13:49, Manish G wrote: > Hi, > > Is it possible to log Flink metrics in application logs apart from > publishing it to Prometheus? > > With regards

Re: Logging Flink metrics

ttps://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: > Hi, > > Is it possible to log Flink metrics in application logs apart from

Re: Logging Flink metrics

; Have you looked at the SLF4J reporter? >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter >> >> On 06/07/2020 13:49, Manish G wrote: >> > Hi, >> > >> > Is it possible to log Flink metrics in application logs apart from >> > publishing it to Prometheus? >> > >> > With regards >> >> >> >

Re: Logging Flink metrics

SLF4J reporter? https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: > Hi, > > Is it possible to log Flink metrics in application logs apart from > p

Re: Logging Flink metrics

4jreporter > > On 06/07/2020 13:49, Manish G wrote: > > Hi, > > > > Is it possible to log Flink metrics in application logs apart from > > publishing it to Prometheus? > > > > With regards > > >

Re: Logging Flink metrics

Have you looked at the SLF4J reporter? https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter On 06/07/2020 13:49, Manish G wrote: Hi, Is it possible to log Flink metrics in application logs apart from publishing it

Logging Flink metrics

Hi, Is it possible to log Flink metrics in application logs apart from publishing it to Prometheus? With regards

Re: Re: How to dynamically initialize flink metrics in invoke method and then reuse it？

nd get the > corresponding Meter according to tableName in the `invoke` method. > > > Thanks, > Lei > -- > wangl...@geekplus.com.cn > > > *Sender:* wangl...@geekplus.com.cn > *Send Time:* 2020-07-03 14:27 > *Receiver:* Xintong Song > *cc:

Re: Re: How to dynamically initialize flink metrics in invoke method and then reuse it？

wangl...@geekplus.com.cn Send Time: 2020-07-03 14:27 Receiver: Xintong Song cc: user Subject: Re: Re: How to dynamically initialize flink metrics in invoke method and then reuse it？ Hi Xintong, Yes, initializing the metric in the `open` method works, but it doesn't solve my problem. I want to init

Re: Re: How to dynamically initialize flink metrics in invoke method and then reuse it？

inlog info. I want to monitor the qps by tableName. The tableName is different for every record. Thanks, Lei wangl...@geekplus.com.cn Sender: Xintong Song Send Time: 2020-07-03 13:14 Receiver: wangl...@geekplus.com.cn cc: user Subject: Re: How to dynamically initialize flink metrics in i

Re: How to dynamically initialize flink metrics in invoke method and then reuse it？

.cn> wrote: > > In one flink operator, i want to initialize multiple flink metrics > according to message content. > As the code below. > > public void invoke(ObjectNode node, Context context) throws Exception { > > String tableName = node.get("metadata").

How to dynamically initialize flink metrics in invoke method and then reuse it？

In one flink operator, i want to initialize multiple flink metrics according to message content. As the code below. public void invoke(ObjectNode node, Context context) throws Exception { String tableName = node.get("metadata").get("topic").asText(); Meter meter

Re: Flink Metrics in kubernetes

Hi Gary, Sorry for the false alarm. It's caused by a bug in my deployment - no metrics were added into the registry. Sorry for wasting your time. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink Metrics in kubernetes

Hi Gary, Thanks for the help. Here below is the output from jstack. It seems not being blocked. In my JobManager log, there's this WARN, I am not sure whether it's relevant at all. Attached is the full jstack dump k8xDump.txt

Re: Flink Metrics in kubernetes

t; to NewRelic (using a custom ScheduledDropwizardReporter). > > From the logs, I could see that an instance of ScheduledDropwizardReporter > has already been created successfully (the overridden getReporter() method > <https://github.com/apache/flink/blob/e346215edcf2252cc60c5ce

Flink Metrics in kubernetes

) method <https://github.com/apache/flink/blob/e346215edcf2252cc60c5cef507ea77ce2ac9aca/flink-metrics/flink-metrics-dropwizard/src/main/java/org/apache/flink/dropwizard/ScheduledDropwizardReporter.java#L234> was called). An instance of MetricRegistryImpl <https://github.com/

Re: Flink Metrics - PrometheusReporter

[emailsignature] From: Chesnay Schepler Sent: Wednesday, January 22, 2020 6:07 PM To: Sidney Feiner ; flink-u...@apache.org Subject: Re: Flink Metrics - PrometheusReporter Metrics are exposed via reporters by each process separately, whereas the WebUI aggregates metrics. As such you ha

Re: Flink Metrics - PrometheusReporter

ed in locally on my computer, I would access the port I configured and see all the metrics I've created. In production, we use High Availability mode and when I try to access the JobManager's metrics in the port I've configured on the PrometheusReporter, I see some very basic met

Flink Metrics - PrometheusReporter

x27;ve configured on the PrometheusReporter, I see some very basic metrics - default Flink metrics, but I can't see any of my custom metrics. Weird thing is I can see those metrics through Flink's UI in the Metrics tab: [cid:dc6050e2-a947-4856-8339-5daea66b6a77] Does anybody have a

Re: Does Flink Metrics provide information about each records inserted into the database

What about using an accumulator? Does it work for you needs? Il Sab 18 Gen 2020, 10:03 Soheil Pourbafrani ha scritto: > Hi, > > I'm using Flink to insert some processed records into the database. I need > to have some aggregated information about records inserted into the > database so far. For

Does Flink Metrics provide information about each records inserted into the database

Hi, I'm using Flink to insert some processed records into the database. I need to have some aggregated information about records inserted into the database so far. For example, for a specific column value, I need to know how many records have been inserted. Can I use the Flink Matrics to provide t

Re: Apache Flink - Flink Metrics collection using Prometheus on EMR from streaming mode

ecommended for a streaming job？ Best,Vino M Singh 于2019年12月24日周二 下午4:02写道： Hi: I wanted to find out what's the best way of collecting Flink metrics using Prometheus in a streaming application on EMR/Hadoop. Since the Flink streaming jobs could be running on any node - is there any

Re: Apache Flink - Flink Metrics collection using Prometheus on EMR from streaming mode

aming job？ > > Best, > Vino > > M Singh 于2019年12月24日周二 下午4:02写道： > >> Hi: >> >> I wanted to find out what's the best way of collecting Flink metrics >> using Prometheus in a streaming application on EMR/Hadoop. >> >> Since the Flink streami

Re: Apache Flink - Flink Metrics collection using Prometheus on EMR from streaming mode

treaming job？ Best, Vino M Singh 于2019年12月24日周二 下午4:02写道： > Hi: > > I wanted to find out what's the best way of collecting Flink metrics using > Prometheus in a streaming application on EMR/Hadoop. > > Since the Flink streaming jobs could be running on any node - is ther

Apache Flink - Flink Metrics collection using Prometheus on EMR from streaming mode

Hi: I wanted to find out what's the best way of collecting Flink metrics using Prometheus in a streaming application on EMR/Hadoop. Since the Flink streaming jobs could be running on any node - is there any Prometheus configuration or service discovery option available that will dynami

Re: Apache Flink - Flink Metrics - How to distinguish b/w metrics for two job manager on the same host

Thanks Vino and Biao for your help.  Mans On Thursday, December 19, 2019, 02:25:40 AM EST, Biao Liu wrote: Hi Mans, That's indeed a problem. We have a plan to fix it. I think it could be included in 1.11. You could follow this issue [1] to check the progress. [1] https://issues.apache

Re: Apache Flink - Flink Metrics - How to distinguish b/w metrics for two job manager on the same host

Hi Mans, That's indeed a problem. We have a plan to fix it. I think it could be included in 1.11. You could follow this issue [1] to check the progress. [1] https://issues.apache.org/jira/browse/FLINK-9543 Thanks, Biao /'bɪ.aʊ/ On Thu, 19 Dec 2019 at 14:51, vino yang wrote: > Hi Mans, > > I

Re: Apache Flink - Flink Metrics - How to distinguish b/w metrics for two job manager on the same host

Hi Mans, IMO, one job manager represents one Flink cluster and one Flink cluster has a suite of Flink configuration e.g. metrics reporter. Some metrics reporters support tag feature, you can specify it to distinguish different Flink cluster.[1] [1]: https://ci.apache.org/projects/flink/flink-doc

Apache Flink - Flink Metrics - How to distinguish b/w metrics for two job manager on the same host

Hi: I am using AWS EMR with Flink application and two of the job managers are running on the same host.  I am looking at the metrics documentation (Apache Flink 1.9 Documentation: Metrics) and and see the following:  | | | | Apache Flink 1.9 Documentation: Metrics | | | - metr

Re: Different jobName per Job when reporting Flink metrics to PushGateway

> the code is ignored. > > So how can I change the jobName per job? And if I can't, is there a way to > set additional Labels when reporting the metrics? Because I haven't seen an > option for that as well. > > Thanks :) > > > I've posted this on Sta

Different jobName per Job when reporting Flink metrics to PushGateway

t as well. Thanks :) I've posted this on StackOverflow as well - here<https://stackoverflow.com/questions/59376693/different-jobname-per-job-when-reporting-flink-metrics-to-pushgateway> :) Sidney Feiner / Data Platform Developer M: +972.528197720 / Skype: sidney.feiner.startapp [emailsignature]

Re: Flink metrics reporters documentation

Wouldn't be better to provide a link >> to >>the jar directly (on Maven Central for example)? >>2. If you look to influxdb dependency the documentation says to use >>flink-metrics-influxdb-1.9.0.jar but there's no such "unified" jar, on &g

Re: Flink metrics reporters documentation

e jars from /opt >folder..obviously this is not true. Wouldn't be better to provide a link to >the jar directly (on Maven Central for example)? >2. If you look to influxdb dependency the documentation says to use >flink-metrics-influxdb-1.9.0.jar but there's no such

Flink metrics reporters documentation

ovide a link to the jar directly (on Maven Central for example)? 2. If you look to influxdb dependency the documentation says to use flink-metrics-influxdb-1.9.0.jar but there's no such "unified" jar, on maven central there are two version: 1 for scala 2.11 and one for

Re: Flink metrics scope for YARN single job

Hi Biao! > Do you mean "distinguish metrics from different JobManager running on same host"? Exactly. >Will give you a feedback if there is a conclusion. Thanks! On Thu, 15 Aug 2019 at 06:40, Biao Liu wrote: > Hi Vasily, > > > Is there any way to distinguish logs from different JobManager r

Re: Flink metrics scope for YARN single job

Hi Vasily, > Is there any way to distinguish logs from different JobManager running on same host? Do you mean "distinguish metrics from different JobManager running on same host"? I guess there is no other variable you could use for now. But I think it's reasonable to support this requirement. I

Flink metrics scope for YARN single job

Hi, I want to run Flink apps on YARN in single job mode and keep metrics in Graphite. But as i see, the only variable i can use for JobManager scope customization is : https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#list-of-all-variables Is there any way to distingui

Fwd: Flink metrics with parallel operator

From: Sibendu Dey Date: Thu, Aug 1, 2019 at 2:42 AM Subject: Flink metrics with parallel operator To: Hello, I have been working on a flink project and need some help with the metric system. I have a logic inside a process function which side outputs a particular message on certain fa

Re: Flink Metrics

Hi Brian, You can implement a new org.apache.flink.metrics.reporter.MetricReporter as you like and register it to flink in flink conf. e.g. metrics.reporters:my_reporter metrics.reporter.my_other_reporter.class: xxx metrics.reporter.my_other_reporter.config1: yyy metrics.reporter.my_other_report

Flink Metrics

Hi, I am trying to profile my Flink job. For example I want to output the results of the TaskIOMetricGroup to a log file. Does anyone know if there is a way to access this object at runtime and execute the methods to get the data from within my user code that I submit to the Flink to start a j

Re: Flink metrics missing from UI 1.7.2

Aha! This is almost certainly it. I remembered thinking something like this might be a problem. I'll need to change the deployment a bit to add this (not straightforward to edit the YAML in my case, but thanks! On Sun, Mar 24, 2019 at 10:01 AM dawid < apache-flink-user-mailing-list-arch...@davidha

Re: Flink metrics missing from UI 1.7.2

Padarn Wilson-2 wrote > I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job > manager separate. > > I'm having trouble seeing the metrics from my Flink job in the UI > dashboard. Actually I'm using the Datadog reporter to expose most of my > metrics, but latency tracking doe

Re: Flink metrics missing from UI 1.7.2

Thanks David. I cannot see the metrics there, so let me play around a bit more and make sure they are enabled correctly. On Sat, Mar 23, 2019 at 9:19 PM David Anderson wrote: > > I have done this (actually I do it in my flink-conf.yaml), but I am not > seeing any metrics at all in the Flink UI,

Re: Flink metrics missing from UI 1.7.2

> I have done this (actually I do it in my flink-conf.yaml), but I am not seeing any metrics at all in the Flink UI, > let alone the latency tracking. The latency tracking itself does not seem to be exported to datadog (should it be?) The latency metrics are job metrics, and are not shown in the F

Re: Flink metrics missing from UI 1.7.2

Because latency tracking is expensive, it is turned off by default. You turn it on by setting the interval; that looks something like this: env.getConfig().setLatencyTrackingInterval(1000); The full set of configuration options is described in the docs: https://ci.apache.org/projects/flink/fl

Flink metrics missing from UI 1.7.2

Hi User, I am running Fink 1.7.2 on Kubernetes in a setup with task manager and job manager separate. I'm having trouble seeing the metrics from my Flink job in the UI dashboard. Actually I'm using the Datadog reporter to expose most of my metrics, but latency tracking does not seem to be exporte

Re: EOF on scraping flink metrics

Thank you, This is following https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter . What might I be doing wrong ? metrics.reporters: prom metrics.reporter.prom.port: 9610 . metrics.reporter.pr

Re: EOF on scraping flink metrics

Since you're using Prometheus I would recommend setting up a PrometheusReporter as described in the metrics documentation and scrape each JM/TM individually. Scraping through the REST API is more expensive and you loose out on a lot of features. The REST API calls are primarily aimed at the Web

EOF on scraping flink metrics

A simple query, Is the route to /metrics execute an access to an in memory registry of stats collected OR does it contend with access from JM or do expensive access or computation. I see occasionally our Prometheus scrape fail with the error pasted below. We have had the scrapper do much more el

Re: Flink metrics in kubernetes deployment

If you're working with 1.7/master you're probably running into https://issues.apache.org/jira/browse/FLINK-11127 . On 17.12.2018 18:12, eric hoffmann wrote: Hi, In a Kubernetes delpoyment, im not able to display metrics in the dashboard, I try to expose and fix the metrics.internal.query-servi

  1   2   >