or your watermark
> strategy setup?
>
>
>
> Just as said before, ignoring-late-events is a default strategy, that can
> be adjusted by means of a custom window trigger which trades off between
> latency, state size, correctness of the final results.
>
>
>
> Thi
.
Thias
From: John Smith
Sent: Freitag, 26. November 2021 17:17
To: Schwalbe Matthias
Cc: Caizhi Weng ; user
Subject: Re: Windows and data loss.
Or as an example we have a 5 minutes window and lateness of 5 minutes.
We have the following events in the logs
10:00:01 PM > Already pushed
ook [1] and the excerpt
>> blog: [2] [3]
>>
>> I believe his code uses Beam, but the same ideas can be implemented
>> directly in Flink API
>>
>>
>>
>> [1] https://www.oreilly.com/library/view/streaming-systems/9781491983867/
>>
>> [2] ht
gs break down
> (network interrupt, datacenter flooded etc. 😊). With processing time
> events count into the time window when processed, with event time they
> count into the time window when originally created a the source … even if
> processed much later …
>
>
>
> Thias
] https://www.oreilly.com/radar/the-world-beyond-batch-streaming-102/
… happy to discuss further 😊
Thias
From: John Smith
Sent: Freitag, 26. November 2021 14:09
To: Schwalbe Matthias
Cc: Caizhi Weng ; user
Subject: Re: Windows and data loss.
But if we use event time, if a failure happens
gt;
>
>
>
> *From:* John Smith
> *Sent:* Freitag, 26. November 2021 02:55
> *To:* Schwalbe Matthias
> *Cc:* Caizhi Weng ; user
> *Subject:* Re: Windows and data loss.
>
>
>
> Well what I'm thinking for 100% accuracy no data loss just to base the
> count on p
:55
To: Schwalbe Matthias
Cc: Caizhi Weng ; user
Subject: Re: Windows and data loss.
Well what I'm thinking for 100% accuracy no data loss just to base the count on
processing time. So whatever arrives in that window is counted. If I get some
events of the "current" window late a
long
>you will extend the window processing (or is that done in the trigger … I
>don’t remember right know)
>- overall window state grows, if you extend window processing to after
>it is finished …
>
>
>
> Hope this helps 😊
>
>
>
> Thias
>
>
&g
: John Smith
Cc: user
Subject: Re: Windows and data loss.
Hi!
Are you using the datastream API or the table / SQL API? I don't know if
datastream API has this functionality, but in table / SQL API we have the
following configurations [1].
* table.exec.emit.late-fire.enabled: Emit w
Thanks. Using, data streaming.
On Wed, 24 Nov 2021 at 20:56, Caizhi Weng wrote:
> Hi!
>
> Are you using the datastream API or the table / SQL API? I don't know if
> datastream API has this functionality, but in table / SQL API we have the
> following configurations [1].
>
>- table.exec.emit.
Hi!
Are you using the datastream API or the table / SQL API? I don't know if
datastream API has this functionality, but in table / SQL API we have the
following configurations [1].
- table.exec.emit.late-fire.enabled: Emit window results for late
records;
- table.exec.emit.late-fire.dela
Hi I understand that when using windows and having set the watermarks and
lateness configs. That if an event comes late it is lost and we can
output it to side output.
But wondering is there a way to do it without the loss?
I'm guessing an "all" window with a custom trigger that just fires X peri
12 matches
Mail list logo
