Re: Adding custom monitoring to Flink

2016-04-19 Thread Till Rohrmann
Hi Maxim, I think the corresponding JIRA issue is https://issues.apache.org/jira/browse/FLINK-456 Cheers, Till On Thu, Apr 14, 2016 at 10:50 PM, Maxim wrote: > I don't have full list of metrics, but everything that is related to > runtime performance and possible bottlenecks of the system. All

Re: Adding custom monitoring to Flink

2016-04-14 Thread Maxim
I don't have full list of metrics, but everything that is related to runtime performance and possible bottlenecks of the system. All interprocess communication counters, errors, latencies, checkpoint sizes and checkpointing latencies. Buffer allocations and releases, etc. As we aggregate ourselves

Re: Adding custom monitoring to Flink

2016-04-14 Thread Chesnay Schepler
I'm currently working on a metric system that a) exposes several TaskManger metrics b) allows gathering metrics in various parts of a task, most notably user-defined functions. The first version makes these metrics available via JMX on each TaskManager. While a mechanism to make that pluggable

Adding custom monitoring to Flink

2016-04-14 Thread Maxim
Hi! I'm looking into integrating Flink into our stack and one of the requirements is to report metrics to an internal system. The current Accumulators are not adequate to provide visibility that we need to run such a system in production. We want much more information about the internal cluster sta