Re: Watermark Alignment on Flink Runner's UnboundedSourceWrapper

2023-05-23 Thread Jan Lukavský
Yes, FlinkRunner supports Beam's event-time semantics without any additional configuration options.  Jan On 5/23/23 09:52, Talat Uyarer via dev wrote: Hi Jan, Yes My plan is implementing this feature on FlinkRunner. I have one more question. Does Flink Runner support EventTime or Beam  Cust

Re: Watermark Alignment on Flink Runner's UnboundedSourceWrapper

2023-05-23 Thread Talat Uyarer via dev
Hi Jan, Yes My plan is implementing this feature on FlinkRunner. I have one more question. Does Flink Runner support EventTime or Beam Custom Watermark ? Do I need to set AutoWatermarkInterval for stateful Beam Flink Jobs. Or Beam timers can handle it without setting that param ? Thanks On Tue,

Re: Watermark Alignment on Flink Runner's UnboundedSourceWrapper

2023-05-23 Thread Jan Lukavský
Hi Talat, your analysis is correct, aligning watermarks for jobs with high watermark skew in input partitions really results in faster checkpoints and reduces the size of state. There are generally two places you can implement this - in user code (the source) or inside runner. The user code c