Re: Watermarking in Src and Timestamping downstream

2019-08-12 Thread Roshan Naik
Thanks Stephan. It was the case,.. I had an empty override of the processWatermark() in the operator that went unnoticed. Removing it fixed the problem.-roshan On Monday, August 12, 2019, 02:39:45 AM PDT, Stephan Ewen wrote: Do you know what part of the code happens to block off your

Re: Watermarking in Src and Timestamping downstream

2019-08-12 Thread Stephan Ewen
Do you know what part of the code happens to block off your watermark? Maybe a method that is overridden in AbstractStreamOperator in your code? On Sat, Aug 10, 2019 at 4:06 AM Roshan Naik wrote: > Have streaming use cases where it is useful & easier to generate the > watermark in the Source (vi

Watermarking in Src and Timestamping downstream

2019-08-09 Thread Roshan Naik
Have streaming use cases where it is useful & easier to generate the watermark in the Source (via ctx.emitWatermark() ) and assign timestamps in a downstream custom operator which calls  output.collect(new StreamRecord(msg, time)). When doing so, I see that the watermark reaches the downstream op