Hi,
yes AJ that observation is correct. Let's see what Shannon has to say about
this but it might be that all "higher-level" aggregates will have to be
based on the first level and can then update at the speed of that aggregate.
Cheers,
Aljoscha
On Mon, 12 Sep 2016 at 05:03 aj.h wrote:
> In the
In the way that FLIP-2 would solve this problem, secondAggregate would ignore
the early firing updates from firstAggregate to prevent double-counting,
correct? If that's the case, I am trying to understand why we'd want to
trigger early-fires every 30 seconds for the secondAggregate if it's only
ac
Hi,
Sorry, I was little busy lately.
I have most part of FLIP-4 ready except ProcessingTimeEvictor case. I was
trying to find the best way to set the ProcessingTime as per the suggestion
from Aljoscha then I got busy with some other work.
We can definitely work together on that.
Thanks
Vishnu
O
Hi,
I'd be very happy to give you pointers for FLIP-2 and FLIP-4. Why don't you
start a separate thread on the dev list so that we don't hijack this thread.
For FLIP-4 we also have to coordinate with Vishnu, he was driving FLIP-4
but lately everyone has been a bit inactive on that. Let's see if he
Hi, I'm interested in helping out on this project. I also want to implement a
continuous time-boxed sliding window, my current use case is a 60-second
sliding window that moves whenever a newer event arrives, discarding any
late events that arrive outside the current window, but *also* re-triggerin
h can have an impact on the processing guarantees
> when a failure/recovery occurs if we don't do it carefully. Also, we're not
> particularly sophisticated yet with regard to avoiding unnecessary queries
> to the time series data.
>
> -Shannon
>
>
> From: Aljoscha K
-Shannon
From: Aljoscha Krettek mailto:aljos...@apache.org>>
Date: Friday, September 2, 2016 at 4:02 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Firing windows multiple times
I see, I didn't forget abo
ll
> be on average 0.5 months stale. A year-long window is even worse.
>
> -Shannon
>
> From: Aljoscha Krettek
> Date: Tuesday, August 30, 2016 at 9:08 AM
> To: Shannon Carey , "user@flink.apache.org" <
> user@flink.apache.org>
>
> Subject: Re: Firing
ia.com>>,
"user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Firing windows multiple times
Hi,
I think this can be neatly expressed by using something like a tree of windowed
aggregations, i.e. you specify your smallest window c
rent watermark) on an ongoing basis.
> The windowing function would be responsible for evicting old data based on
> the current watermark.
>
> Does that make sense? Does it seem logical, or am I misunderstanding
> something about how Flink works?
>
> -Shannon
>
>
> Fr
ljoscha Krettek mailto:aljos...@apache.org>>
Date: Monday, August 29, 2016 at 3:56 AM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Firing windows multiple times
Hi,
that would certainly be possible? What do you th
ark to the window
> function metadata in FLIP-2?
>
> From: Shannon Carey
> Date: Friday, August 12, 2016 at 6:24 PM
> To: Aljoscha Krettek , "user@flink.apache.org" <
> user@flink.apache.org>
>
> Subject: Re: Firing windows multiple times
>
> Thanks Al
ser@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Firing windows multiple times
Thanks Aljoscha, I didn't know about those. Yes, they look like handy changes,
especially to enable flexible approaches for eviction. In particular, having
the current watermark availab
.org>"
mailto:user@flink.apache.org>>
Subject: Re: Firing windows multiple times
Hi,
there is already this FLIP:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor
which also links to a mailing list discussion. And this FLIP:
https://cwiki.apache.or
Hi,
there is already this FLIP:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor
which
also links to a mailing list discussion. And this FLIP:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-2+Extending+Window+Function+Metadata.
The former proposes to enhan
"If Window B is a Folding Window and does not have an evictor then it should
not keep the list of all received elements."
Agreed! Upon closer inspection, the behavior I'm describing is only present
when using EvictingWindowOperator, not when using WindowOperator. I misread
line 382 of WindowOpe
ermark, trigger's event timer is reached,
>fires and purges and emits current state as event z(time=1, count=2)
>9. Window B receives event, trigger waits for processing time delay,
>then executes fold() and emits event(time=1 => count=2), but internal
>Window stat
current state as event z(time=1, count=2)
>> Window B receives event, trigger waits for processing time delay, then
>> executes fold() and emits event(time=1 => count=2), but internal Window
>> state looks like [x(time=1, count=1), y(time=1, count=2), z(time=1, count=2)]
&
t; Does that explanation help interpret my original email?
>
> -Shannon
>
>
> From: Aljoscha Krettek mailto:aljos...@apache.org>>
> Date: Wednesday, August 10, 2016 at 12:18 PM
> To: "user@flink.apache.org <mailto:user@flink.apache.org>"
> mailto:user@
planation help interpret my original email?
-Shannon
From: Aljoscha Krettek mailto:aljos...@apache.org>>
Date: Wednesday, August 10, 2016 at 12:18 PM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Re: Firing windows
Hi Aljoscha,
This looks like the bug that we discussed, as part of Enhance window
evictor JIRA
Thanks,
Vishnu
On Wed, Aug 10, 2016 at 1:18 PM, Aljoscha Krettek
wrote:
> Hi,
> from your mail I'm gathering that you are in fact using an Evictor, is
> that correct? If not, then the window operator
Hi,
from your mail I'm gathering that you are in fact using an Evictor, is that
correct? If not, then the window operator should not keep all the elements
ever received for a window but only the aggregated result.
Side note, there seems to be a bug in EvictingWindowOperator that causes
evicted ele
One unfortunate aspect of using a fold() instead of a window is that the fold
function has no knowledge of the watermarks. As a result, it is difficult to
ensure that only items before the current watermark are included in the
aggregation, and that old items are evicted correctly. This fact lend
23 matches
Mail list logo
