Hi,
I'm trying to understand how the lifecycle of messages / state is managed
by Flink, but I'm failing to find any documentation.
Specially, if I'm using a windowed stream and a type of trigger that retain
the elements of the window to allow for processing of late data e.g.
ContinousEventTimeTri
a new result is emitted.
> */
>public static EventTimeTrigger accumulating(AbstractTime
> allowedLateness) {
> return new EventTimeTrigger(true, allowedLateness.toMilliseconds());
>}
> }
>
> You can specify a lateness and while that time is not yet reached the
> fires. Only after the
> > * watermark passes the specified lateness are the window elements
> discarded, without
> > * emitting a new result. If a late element arrives within the
> specified lateness
> > * the window is computed again and a new result is emit
is smart enough, configurable enough,
or customisable enough to allow *where *the ageing state is kept.
Thoughts?
Thanks!
Andy
On Fri, 15 Jan 2016 at 15:51 Andrew Coates
wrote:
> Hi Aljoscha,
>
> Thanks for the info!
>
> Andy
>
> On Fri, 15 Jan 2016 at 10:12 Aljoscha Krettek wrote:
on s state backend
> that stores state in HDFS TFiles (if I’m not mistaken) and he also
> previously contributed the DB state backend that can store state in a SQL
> data base.
>
> Cheers,
> Aljoscha
>
> On 15 Jan 2016, at 16:56, Andrew Coates wrote:
> >
> >
> >
Hi Stephan,
If we were to do that, would flink leverage the fact that Kafka has already
partitioned the data by the key, or would flink attempt to shuffle the data
again into its own partitions, potentially shuffling data between machines
for no gain?
Thanks,
Andy
On Sun, 10 Apr 2016, 13:22 Ste
Extending on what Henry is asking... What if data can be more that a day
late, or in a more streaming nature, what if updates can come through for
previous values?
This would obviously involve storing a great deal of state. The use case
I'm thinking of has large large volumes per day. So an extern
