Re: Proper way to establish bucket counts

2017-08-04 Thread Fabian Hueske
Hi Robert, That's right. The count's are on a per operator-level. I think you can get down to the task-level but counts per bucket are not tracked. Maybe Chesnay (in CC) can help here. He knows the metrics system the best. @Chesnay, is there a way to expire metric counters? Alternatively, you cou

Re: Proper way to establish bucket counts

2017-08-02 Thread Fabian Hueske
Hi Robert, Flink collects many metrics by default, including the number of records / events that go into each operator (see [1], System Metrics, IO, "numRecordsIn"). So, you would only need to access that metric. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/monito

Proper way to establish bucket counts

2017-08-01 Thread Robert Rapplean
I want a count of events that are put into a bucketing sink, but can't find a ready-made way of doing that. Is there an easier way than to implement a counter for each bucket via the metrics? If metrics counters is the easy way, what do I do to make sure that I don't have a memory leak from expired