@Alexander Sorry about that, that would be my mistake. I’ll close FLINK-9204 as a duplicate and leave my thoughts on FLINK-9155. Thanks for pointing out!
On 19 April 2018 at 2:00:51 AM, Elias Levy (fearsome.lucid...@gmail.com) wrote: Either proposal would work. In the later case, at a minimum we'd need a way to identify the source within the metric. The basic error metric would then allow us to go into the logs to determine the cause of the error, as we already record the message causing trouble in the log. On Mon, Apr 16, 2018 at 4:42 AM, Fabian Hueske <fhue...@gmail.com> wrote: Thanks for starting the discussion Elias. I see two ways to address this issue. 1) Add an interface that a deserialization schema can implement to register metrics. Each source would need to check for the interface and call it to setup metrics. 2) Check for null returns in the source functions and increment a respective counter. In both cases, we need to touch the source connectors. I see that passing information such as topic name, partition, and offset are important debugging information. However, I don't think that metrics would be good to capture them. In that case, log files might be a better approach. I'm not sure to what extend the source functions (Kafka, Kinesis) support such error tracking. Adding Gordon to the thread who knows the internals of the connectors.
