Re: Watermarks and Kafka

2019-07-08 Thread Juan Gentile
the watermarks handled in the source operator. Please let us know your opinion. Thank you, Juan G. From: Konstantin Knauf Date: Sunday, July 7, 2019 at 10:14 PM To: Juan Gentile Cc: "user@flink.apache.org" , Olivier Solliec , Oleksandr Nitavskyi Subject: Re: Watermarks and Kafka H

Re: Watermarks and Kafka

2019-07-07 Thread Konstantin Knauf
Hi Juan, I just replied to your other question, but I think, I better get where you are coming from now. Are you aware of per-partition watermarking [1]? You don't need to manage this map yourself. BUT: this does not solve the problem, that this Map is not stored in Managed State. Watermarks are

Watermarks and Kafka

2019-07-03 Thread Juan Gentile
Hello! We currently have a job which reads from Kafka and uses punctuated watermarks based on the messages we read. We currently keep track of the watermarks for each partition to emit a consensus watermark, taking the smallest of all partitions. We ran into an issue because we are not storing