Re: global watermark across multiple kafka consumers

2015-12-17 Thread Stephan Ewen
@Andrew: Just to make sure that there is no confusion: Even though every Kafka Source generates a local watermark, these watermarks are merged when the streams are merged, for example in union() or keyBy() steps. The operator that merges streams tracks the current watermark of each stream and then

Re: global watermark across multiple kafka consumers

2015-12-16 Thread Till Rohrmann
Hi Andrew, as far as I know, there is nothing such as a prescribed way of handling this kind of situation. If you want to synchronize the watermark generation given a set of KafkaConsumers you need some kind of ground truth. This could be, for example, a central registry such as ZooKeeper in whic