Hi Fabian, We did need a consistent view of data, we need the Counter and HDFS file to be consistent. For example, when the Counter indicate there is 1000 message wrote to the HDFS, there must be exactly 1000 messages in HDFS ready for read.
The data we write to HDFS is collected by an Agent(which also send Counter message to count message number received), data has a timestamp and we use BucktingSink to write data into different bucket. Could you give me a clue on how to achieve this with watermark. As my understanding, watermark is designed to process out-of-order data with a know delay, how it can be used to make my CounterSink and BuckingSink consistent. Thanks, Xilang -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
