Help needed for passing in multiple filters in filter.includes parameter for my flink metric reporter in flink-conf.yaml?

Hi, can you please look at my StackOverflow query? https://stackoverflow.com/questions/76949195/how-to-include-multiple-filters-in-filter-includes-parameter-for-my-flink-metric Thanks!

Flink Metric Naming And Reporting Confusion

Hi, My team just started coding a new Flink app to be deployed under AWS EMR. We are a little confused by the metric naming for built in metrics (not ones we create ourselves) and reporting of metrics via StatsD. We have added configuration to flink-conf.yaml to configure a StatsD reporter. This

Re: Flink metric

Hi, multiple records are in the system at the same time, because Flink is buffering records in various components, for efficiency reasons. That's why you see that an individual record might have a latency of ~100ms, while Flink is processing many more messages. On Tue, Apr 5, 2022 at 12:54 PM lo

Flink metric

Hi,I'm trying to work with monitoring a job on Flink. Just a simple process: a data source from kafka joining with a data stream table (load from database).I've already enabled the latency tracking like the instruction from:  https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/met

Re: Flink Metric Reporting from Job Manager

Hi, I think that is not directly supported. After all, the main method can also be executed outside of a JobManager and there you don't have any Flink context/connections/components set up. Best, Dawid On 08/07/2021 00:12, Mason Chen wrote: > Hi all, > > Does Flink support reporting metrics fro

Flink Metric Reporting from Job Manager

Hi all, Does Flink support reporting metrics from the main method that is ran on the Job Manager? In this case, we want to report a failure to add an operator to the Job Graph. Best, Mason

Re: Flink Metric isBackPressured not available

The isBackPressured metric is a Boolean -- it reports true or false, rather than 1 or 0. The Flink web UI can not display it (it shows NaN); perhaps the same is true for Datadog. https://issues.apache.org/jira/browse/FLINK-15753 relates to this. Regards, David On Tue, Apr 13, 2021 at 12:13 PM Cl

Re: Flink Metric isBackPressured not available

Thanks for your reply. I'm using Flink 1.12. I'm checking in Datadog and the metric is not available there. It has other task/operator metrics such as numRecordsIn/numRecordsOut there but not the isBackPressured. On Mon, Apr 12, 2021 at 8:40 AM Roman Khachatryan wrote: > Hi, > > The metric is

Re: Query regarding flink metric types

Hi Suchithra, You are right, those metrics can only grow, at least until failover. isBackPressured is reported as a boolean on subtask level. These samples are then aggregated and a ratio of (times-back-pressured / number-of-samples) is reported to the JobManager. Regards, Roman On Fri, Apr 9,

Re: Flink Metric isBackPressured not available

Hi, The metric is registered upon task deployment and reported periodically. Which Flink version are you using? The metric was added in 1.10. Are you checking it in the UI? Regards, Roman On Fri, Apr 9, 2021 at 8:50 PM Claude M wrote: > > Hello, > > The documentation here > https://ci.apache.

Flink Metric isBackPressured not available

Hello, The documentation here https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html states there is a isBackPressured metric available yet I don't see it. Any ideas why? Thanks

Query regarding flink metric types

Hi Community, Need some information regarding metrics type mentioned in flink documentation. https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html For the checkpoint metrics, below metrics are defined as of type gauge. As per my understanding gauge type is used to represent a v

Re: Can back pressure data be gathered by Flink metric system?

-- From:徐涛 Send Time:2019年4月16日(星期二) 09:33 To:zhijiang Cc:user Subject:Re: Can back pressure data be gathered by Flink metric system? Hi Zhijiang, Because I want to know the current and the trend of backpressure status in Flink Job. Like other index such as latency, I can

Re: Can back pressure data be gathered by Flink metric system?

pache.org/projects/flink/flink-docs-release-1.8/monitoring/back_pressure.html > > Best, > Zhijiang > > -- > From:徐涛 > Send Time:2019年4月15日(星期一) 10:19 > To:user > Subject:Can back pressure data be gathered

Re: Can back pressure data be gathered by Flink metric system?

ure.html > > Best, > Zhijiang > > -- > > From:徐涛 > Send Time:2019年4月15日(星期一) 10:19 > To:user > Subject:Can back pressure data be gathered by Flink metric system? > > Hi Experts, > From the page Flink metric system( > https://ci.apache.o

Re: Can back pressure data be gathered by Flink metric system?

-- From:徐涛 Send Time:2019年4月15日(星期一) 10:19 To:user Subject:Can back pressure data be gathered by Flink metric system? Hi Experts, From the page Flink metric system(https://ci.apache.org/projects/flink/flink-docs-release-1.8

Can back pressure data be gathered by Flink metric system?

Hi Experts, From the page Flink metric system(https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#system-metrics <https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/metrics.html#system-metrics>), I do not find the info about th

Re: Flink - Metric are not reported

Yes, you can override notifyOnAddedMetric/notifyOnRemovedMetric to ensure that a metric is logged at least once. On 28.11.2018 16:01, bastien dine wrote: Yea, that was I was thinking.. Batch can be quick Can I report metric on "added" action ? (i should override the notifyOnAddedMetric to repo

Re: Flink - Metric are not reported

Yea, that was I was thinking.. Batch can be quick Can I report metric on "added" action ? (i should override the notifyOnAddedMetric to report ?) -- Bastien DINE Data Architect / Software Engineer / Sysadmin bastiendine.io Le mer. 28 nov. 2018 à 15:54, Chesnay Schepler a écrit

Re: Flink - Metric are not reported

How quick does job batch terminate? Metrics are unregistered once the job ends; if the job duration is shorter than the report interval they may never be exposed. On 28.11.2018 15:18, bastien dine wrote: Hello Chesnay, Thanks for your response ! I have logs enable (info), slf4jReporter is wor

Re: Flink - Metric are not reported

Hello Chesnay, Thanks for your response ! I have logs enable (info), slf4jReporter is working, I can see : 15:16:00.112 [Flink-MetricRegistry-thread-1] INFO org.apache.flink.metrics.slf4j.Slf4jReporter - === Starting metrics report === -- Counters

Re: Flink - Metric are not reported

Please enable WARN logging and check for warnings by the SLF4JReporter and/or MetricQueryService. On 27.11.2018 17:00, bastien dine wrote: Hello everyone, Once again I require your help ! I am trying to report custom metric (see my code below) Yet, I do not see them anywhere.. nor in the metri

Flink - Metric are not reported

Hello everyone, Once again I require your help ! I am trying to report custom metric (see my code below) Yet, I do not see them anywhere.. nor in the metric tab from my tasks, nor in the rest API, nor in the declared slf4j reporter.. Can someone help me to debug this .. Here is my RichMap functio

Re: How to configure the reporting interval for the flink metric monitoring.

You can configure the interval by setting |metrics.reporter..interval| as described in the documentation . On 17.04.2018 13:40, Ganesh Manal wrote: HI, When I configure the counter ( in default system me

How to configure the reporting interval for the flink metric monitoring.

HI, When I configure the counter ( in default system metric ). I could see the counter getting monitored on reporting tool ( graphite in my case ). But the default reporting interval is 60 seconds. Is there a way to configure the interval for metric reporting ? Thanks & Regards, Ganesh Manal

Re: need instruction on how the Flink metric works

I just did the same test as you had with SocketWindowWordCount, and the counter showed up all right. You should probably connect Jconsole to localhost:28781 (or whatever port you have your JMX server listened on) That's how I setup the env, perhaps there is other better ways to do it. On Wed, Se

Re: need instruction on how the Flink metric works

Still got stuck, here are my steps (on my laptop) for example: Step1: public class MetricsTest extends RichMapFunction { private static final long serialVersionUID = 1L; private org.apache.flink.metrics.Meter meter; private Counter counter; @Override public void open(Configurat

Re: need instruction on how the Flink metric works

Hi, There are several possibilities: 1. Please check if reporter is set up ( guide ) For example, I would make sure my local JMXReporter service is up and running by check

Re: need instruction on how the Flink metric works

Thanks, When I started jconsole, it listed com.apache.flink.runtime.jobmanager..:[port] as one of the Local Process, i was able to connect to it with insecure connection, but i was not able to locate the Counter metrics, I only saw some system metrics. On Sun, Sep 17, 2017 at 7:39 PM, Michael Fong

Re: need instruction on how the Flink metric works

Hi, You may enable metrics reporter to see the output of your metrics; counter in your example. There is a brief documentation regarding to metrics and reporter setup at link . The easiest approach, in my opinion

need instruction on how the Flink metric works

I'm new to flink and I have read https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html, I am still unclear where do I read the metrics I added. for example, public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExe