@Sameer, yes, if one source stops emitting watermarks then downstream
operations will buffer data until the source starts updating the watermark
again. If you can live with some data being late you could change the
watermark logic in the source to start advancing the watermark if no new
data is arr
Alijoscha - Thanks it works exactly as you said. I found out why my windows
were firing twice. I was making the error of adding the
AutoWatermarkInterval to the existing watermark each time the watermark was
sampled from the source just to fire a window if one of the sources was
delayed substantial
Aljoscha,
Awesome. Exactly the behavior I was hoping would be exhibited. Thank you
for the quick answer :)
Thanks,
David
On Thu, Jul 21, 2016 at 2:17 AM, Aljoscha Krettek
wrote:
> Hi David,
> windows are being processed in order of their end timestamp. So if you
> specify an allowed lateness o
Stream2 does send watermarks only after it sees elements C,D. It send the
watermark (5) 20 seconds after Stream 1 sends it.
>From what I understand Flink merges watermarks from both streams on the
Reduce side. But does it wait a certain pre-configured amount of time (for
watermarks from both strea
Yes, that is to be expected. Stream 2 should only send the watermark once
the elements with a timestamp lower than the watermark have been sent as
well.
On Thu, 21 Jul 2016 at 13:10 Sameer W wrote:
> Thanks, Aljoscha,
>
> This what I am seeing when I use Ascending timestamps as watermarks-
>
> C
Thanks, Aljoscha,
This what I am seeing when I use Ascending timestamps as watermarks-
Consider a window if 1-5 seconds
Stream 1- Sends Elements A,B
Stream 2 (20 seconds later) - Sends Elements C,D
I see Window (1-5) fires first with just A,B. After 20 seconds Window (1-5)
fires again but this
Hi David,
windows are being processed in order of their end timestamp. So if you
specify an allowed lateness of zero (which will only be possible on Flink
1.1 or by using a custom trigger) you should be able to sort the elements.
The ordering is only valid within one key, though, since windows for
Hi,
If watermarks arriving from multiple sources, how long does the Event Time
Trigger wait for the slower source to send its watermarks before triggering
only from the faster source? I have seen that if one of the sources is really
slow then the elements of the faster source fires and when the
Hi David,
You are right, the events in the window are not sorted according to the
EventTime hence the processing is not done in an increasing order of
timestamp.
As you said, you will have to do the sorting yourself in your window
function to make sure that you are processing the events in order.
Hi all,
In Flink, after setting the time characteristic to event time and properly
assigning timestamps/watermarks, time-based windows will be created based upon
event time. If we need to process events within a window in event time order,
we can sort the windowed values and process as necessar
10 matches
Mail list logo
