Re: AW: Is watermark used by joining two streams

2017-07-31 Thread G.S.Vijay Raajaa
Hi Fabian, Thanks for the reply. I shall try the CoProcessFunction implementation. Currently, I am trying to assign watermark on the keyed stream. Please find a snippet of the code for better understanding; List < String > names = new ArrayList < > (); names.add("stream_a"); names.add("stream_

Re: AW: Is watermark used by joining two streams

2017-07-31 Thread Fabian Hueske
Hi Vijay, there are many ways to implement joins with a stateful CoProcessFunction. It gives you access to the timestamps of records and you can register timers that trigger when a certain time is reached. It is basically up to you how you join and emit data. You can drop late data or emit it. Ple

Re: AW: Is watermark used by joining two streams

2017-07-31 Thread G.S.Vijay Raajaa
My bad. I meant only join. I am currently using keyBy on a timestamp common across the streams. Regards, Vijay Raajaa GS On Mon, Jul 31, 2017 at 1:16 PM, Fabian Hueske wrote: > Hi, > > @Wei: You can implement very different behavior using a CoProcessFunction. > However, if your operator is time

Re: AW: Is watermark used by joining two streams

2017-07-31 Thread Fabian Hueske
Hi, @Wei: You can implement very different behavior using a CoProcessFunction. However, if your operator is time-based, the logical time of the operator will be the minimum time of both streams (time of the "slower" watermark). @Vijay: I did not understand what your requirements are. Do you want

Re: AW: Is watermark used by joining two streams

2017-07-30 Thread G.S.Vijay Raajaa
Hi Fabian, How do I order by the merge time. Let's say I merge the stream at T1. I wanted to drop T2 merge if T2 < T1. Now depending on the arrival of data from individual stream and the time at which the merge happens, they become out of order. Any thoughts will be really appreciated. Regards, V

AW: Is watermark used by joining two streams

2017-07-30 Thread wei
Hello Fabian, thank you for your answer! Does it mean that the operator will wait until get two watermarks from the input streams and emits then the “slower” watermark? Best regards Wei Von: Fabian Hueske [mailto:fhue...@gmail.com] Gesendet: Sunday, July 30, 2017 11:17 AM An: xie