Re: merge/union fast and slow streams based on event timestamp

2018-05-02 Thread Fabian Hueske
Hi Tao, there's no built-in mechanism for this in Flink. Throttling streams and creating back pressure is not a good idea in general because it prevents checkpoint barriers (which must be aligned with the events) to arrive a the operators. This might cause checkpoints to time out. The only way to

merge/union fast and slow streams based on event timestamp

2018-04-30 Thread Tao Xia
I am running into a problem when processing the past 7 days of data from multiple streams. I am trying to union the streams based on event timestamp. The problem is that there are streams are significant big than other streams. For example if one stream has 1,000 event/sec and the other stream ha