Re: Tracing and Flink

2020-08-17 Thread bvarga
Hi Aaron, I've recently been looking at this topic and working on a prototype. The approach I am trying is "backward tracing", or data provenance tracing, where we try to explain what inputs and steps have affected the production of an output record. Arvid has summarized the most important aspect

Re: Tracing and Flink

2020-08-15 Thread Arvid Heise
Hi Aaron, I'm not too sure about tracing and Flink. It's the first time I heard about it in this context and I'm not immediately seeing the benefit of it. What is imho more interesting and a well-formed discipline in the science of data quality is a concept called data lineage

Tracing and Flink

2020-08-14 Thread Aaron Levin
Hello Flink Friends! This is a long-shot, but I'm wondering if anyone is thinking or working on applying tracing to Streaming systems and in particular Flink. As far as I understand this is a fairly open problem and so I'm curious how folks are thinking about it and if anyone has considered how th