Hi, We are looking for a monitoring solution for our dataflow - Track the progress of incoming messages while they are processed. I'll clarify - we want to build some service which will show status for each incoming message. And in case of failures to give some detailed information.
I thought about the following: First, every incoming message will be assigned with some id. We can create a "Reporter" (A logger with some additional capabilities) which each operator can communicate with, and update a status and more relevant information. These details can be stroed in kibana (ES) for example. Then, we need to create another service which will query kibana and shows the results. What do you think about that? Is there any built-in solution for that? (flink built in metrics are not relevant here because they don't help to track a single message) How are you logging and tracking your processed messages? Is there any documentation or some use cases that I can learn from? Thanks, Roey.
