Re: custom metrics in elasticsearch ActionRequestFailureHandler

Hi Alexander, yes in the first iteration the use case is to get visibility on failed ES requests. Usually we expose metrics to count failures and integrate them into dashboards and setup alerting rules which fire in case they hit a certain threshold. In not Flink based applications which index

Re: custom metrics in elasticsearch ActionRequestFailureHandler

Hi Lars, What is your use case for the failure handler, just collecting metrics? We want to remove the configurable failure handler in the new Sink API implementation of the Elasticsearch connector in Flink 1.15 because it can be a huge footgun with regards to delivery guarantees. Best Regards, A

Re: custom metrics in elasticsearch ActionRequestFailureHandler

Hi David, Thanks for the reply. I think especially in an error/failure handler metrics are important in order to have proper monitoring/alerting in such cases. Would be awesome if this could be added to Flink at some point :). Regards, Lars > Am 02.12.2021 um 18:13 schrieb David Morávek : >

Re: custom metrics in elasticsearch ActionRequestFailureHandler

Hi Lars, quickly looking at the ES connector code, I think you're right and there is no way to do that :( In general I'd say that being able to expose metrics is a valid request. I can imagine having some kind of `RichActionRequestFailureHandler` with `{get|set}RuntimeContext` methods. More or l

Re: Custom metrics in Stateful Functions

Hello Cliff, You are right, indeed defining custom metrics is not supported at the moment. I will file a JIRA issue so we can track this, and we will try to prioritize this feature up. Meanwhile, there are a lot of metrics that StateFun defines, like invocations rates etc' perhaps you can find it

Re: custom metrics within a Trigger

Do you mind sharing the code how do you register your metrics with the TriggerContext? It could help us identify where does name collisions come from. As far as I am aware it should be fine to use the TriggerContext for registering metrics. Best, Dawid On 16/03/2021 17:35, Aleksander Sumowski wr

Re: Custom metrics output

Hi Joris, I don't think that the approach of "add methods in operator class code that can be called from the main Flink program" will work. The most efficient approach would be implementing a ProcessFunction that counts in 1-min time buckets (using event-time semantics) and updates the metrics. I

Re: Custom Metrics outside RichFunctions

Thanks for the feedback. I will use elastalert to generate an alarm from the logs. On Wed, Jan 22, 2020, 15:03 Chesnay Schepler wrote: > It is not possible to access metrics from within a schema. > > I can't think of a non-hacky workaround (the hacky one being to create a > custom kafka consumer

Re: Custom Metrics outside RichFunctions

It is not possible to access metrics from within a schema. I can't think of a non-hacky workaround (the hacky one being to create a custom kafka consumer that checks the schema class, casts it to your specific class, and then calls a method on your schema that accepts a metric group). On 22/

Re: Custom Metrics outside RichFunctions

Hi Yun, I'm trying to use inside a custom *DeserializationSchema*. Here is the constructor of *FlinkKafkaConsumer*. Inside *DeserializationSchema* I can't use *getRuntimeContext()*. FlinkKafkaConsumer

Re: Custom Metrics outside RichFunctions

Hi David FlinkKafkaConsumer in itself is RichParallelSourceFunction, and you could call function below to register your metrics group: getRuntimeContext().getMetricGroup().addGroup("MyMetrics").counter("myCounter") Best Yun Tang From: David Magalhães Sent: Tue

Re: Custom Metrics in Windowed AggregateFunction

That was my first alternative actually :) That works well for per window metrics. And though it fits perfectly for smaller windows, it might not be frequent enough for larger window sizes.  Thanks, Chirag  On Wednesday, 19 December, 2018, 4:15:41 PM IST, Dawid Wysakowicz wrote: Hi Chir

Re: Custom Metrics in Windowed AggregateFunction

Hi Chirag, I am afraid you are right you cannot access metrics from within AggregateFunction in WindowedStream. You can though use rich variant of WindowFunction, which is invoked for every window with the results of AggregateFunction. Would that be enough for your use case to use .aggregate(aggre

Re: Custom metrics in Watermark Assigner

Hi Oleksandr, It sounds like a good idea to make a lot of internal information more transparent. But it would be better if you could be more specific. Thanks, vino. Oleksandr Nitavskyi 于2018年9月12日周三 下午9:48写道： > Hello guys, > > > > In our custom AssignerWithPunctuatedWatermarks we want to have

Re: Custom metrics reporter classloading problem

Thanks for the explanation, that makes sense. For some reason I thought that in Yarn all stuff goes into the classpath. Gy Chesnay Schepler ezt írta (időpont: 2018. júl. 11., Sze, 15:16): > Reporters do not have access to libraries provided with user-jars. > They are instantiated when JM/TM sta

Re: Custom metrics reporter classloading problem

Reporters do not have access to libraries provided with user-jars. They are instantiated when JM/TM starts, i.e. before any user-code is even accessible. My recommendation would be to either put the kafka dependencies in the /lib folder or try to relocate the kafka code in the reporter. On 1

Re: Custom Metrics

Hi, > I have couple more questions related to metrics. I use Influx db reporter to > report flink metrics and I see a lot of metrics are bring reported. Is there > a way to select only a subset of metrics that we need to monitor the > application? At this point is up to either reporter, or up

Re: Custom Metrics

Thanks Pitor. I have couple more questions related to metrics. I use Influx db reporter to report flink metrics and I see a lot of metrics are bring reported. Is there a way to select only a subset of metrics that we need to monitor the application? Also, Is there a way to specify custom metics s

Re: Custom Metrics

Hi, Reporting once per 10 seconds shouldn’t create problems. Best to try it out. Let us know if you get into some troubles :) Piotrek > On 11 Dec 2017, at 18:23, Navneeth Krishnan wrote: > > Thanks Piotr. > > Yes, passing the metric group should be sufficient. The subcomponents will > not

Re: Custom Metrics

Thanks Piotr. Yes, passing the metric group should be sufficient. The subcomponents will not be able to provide the list of metrics to register since the metrics are created based on incoming data by tenant. Also I am planning to have the metrics reported every 10 seconds and hope it shouldn't be

Re: Custom Metrics

Hi, I’m not sure if I completely understand your issue. 1. - You don’t have to pass RuntimeContext, you can always pass just the MetricGroup or ask your components/subclasses “what metrics do you want to register” and register them at the top level. - Reporting tens/hundreds/thousands of metric

Re: Custom Metrics Reporting in Flink

This doesn't sound like a proper use-case for the metric system, but for side-outputs with a dedicated sink. My reasoning is that it sounds like you want to push data, whereas the metrics work by drawing a snapshot of all metrics and writing them out, i.e. one value per metric. On 03.10.2017