> a-a
> > >> >> >> > > > >>> (even
> > >> >> >> > > > >>> > before the window & evictor completes) and will be
> > >> looking
> > >> >> for
> > >> >> >&g
t; >> >> > be
> >> >> >> > > > >>> seeing
> >> >> >> > > > >>> >a pattern a-a-b but actual event time is a-b-a then
> >> >> trigger
> >> >> >> > will
> >> >
; -
>> >> >> > > > >>> > only
>> >> >> > > > >>> >when the trigger fires)
>> >> >> > > > >>> >
>> >> >> > > > >>> > Thanks a
> >> >> > > > but
> >> >> > > > >>> also
> >> >> > > > >>> > > keeps some internal state machine to decide when it has
> >> >> > observed
> >> >> > > > the
> >> >> > &g
>>> > sense?
>> >> > > > >>> > >
>> >> > > > >>> > > I'm ignoring time and sorting by time for now because we
>> >> > probably
>> >> > > > >>> need
>> >> > > > >>>
2016 at 23:56 Vishnu Viswanath <
> >> > > > >>> > vishnu.viswanat...@gmail.com
> >> > > > >>> > > >
> >> > > > >>> > > wrote:
> >> > > > >>> > >
> >> > > > >>> > > > Hi,
> >> >
gt; > several
>> > > > use
>> > > > >>> > cases
>> > > > >>> > > > that depend on it.
>> > > > >>> > > >
>> > > > >>> > > > Especially in the case where there ar
gt; > >>> > > evict
> > > > >>> > > > those elements, else(since the elements you are looking for
> > > > haven't
> > > > >>> > > arrived
> > > > >>> > > > yet) wait and try again when
; >
> > > >>> > > >
> > > >>> > > > On Thu, Jul 7, 2016 at 9:19 AM, Radu Tudoran <
> > > >>> radu.tudo...@huawei.com>
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> >
tities
> (evictor,
> > >>> > trigger,
> > >>> > > > > window, apply function). However, I think it's possible just
> to
> > >>> > create
> > >>> > > > more
> > >>> > > >
t;> > > > > evictor ; possibility to delete or mark for eviction elements
> in
> >>> the
> >>> > > > > function...)
> >>> > > > >
> >>> > > > > Regarding the specific case of sorted windows, I
;>> > > iteration X
>>> > > > > all elements were sorted, then in iteration X+1 you will need to
>>> sort
>>> > > > just
>>> > > > > the newly arrived elements (M). I would expect that this number M
>>> >
ould have M * N complexity and
>> if
>> > > M<< N
>> > > > > then the complexity is O(N). Alternatively you can use a binary
>> > search
>> > > > for
>> > > > > insertion and then you further reduce the comple
Big Data Expert
> > > > > IT R&D Division
> > > > >
> > > > >
> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
> > > > > European Research Center
> > > > > Riesstrasse 25, 80992 München
> > > > >
&g
> HUAWEI TECHNOLOGIES Duesseldorf GmbH
> > > > Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
> > > > Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
> > > > Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
> > > > Sitz
orf, Amtsgericht Düsseldorf, HRB 56063,
> > > Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
> > > This e-mail and its attachments contain confidential information from
> > > HUAWEI, which is intended only for the person or entity whose address
> is
> > &
t; scanning
> > all the elements of window while trying to evicting element, which may
> cost
> > many IO operations, such as querying DBs to get elements from state.
> > What's more, when an window aggregation function is invertible, such as
> > sum, which can be u
e intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
>
>
> -Original Message-
> From: 吕文龙(吕文龙) [mailto:wenlong@alibaba-inc.com]
> Sent: Thursday, July 07, 2016 11:59 AM
>
sender by phone or email immediately and
delete it!
-Original Message-
From: 吕文龙(吕文龙) [mailto:wenlong@alibaba-inc.com]
Sent: Thursday, July 07, 2016 11:59 AM
To: dev@flink.apache.org
Subject: 答复: [DISCUSS] Enhance Window Evictor in Flink
HI,
I think it is necessary to support sorted window,
ishes!
---
wenlong.lwl
-邮件原件-
发件人: Aljoscha Krettek [mailto:aljos...@apache.org]
发送时间: 2016年7月7日 17:32
收件人: dev@flink.apache.org
主题: Re: [DISCUSS] Enhance Window Evictor in Flink
Hi,
regarding "sorting the window by event time": I also considered this but in the
end I don't
ut not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
>
> -Original Message-----
&
nath [mailto:vishnu.viswanat...@gmail.com]
Sent: Thursday, July 07, 2016 1:28 AM
To: Dev
Subject: Re: [DISCUSS] Enhance Window Evictor in Flink
Thank you Maxim and Aljoscha.
Yes the beforeEvict and afterEvict should able address point 3.
I have one more use case in my mind (which I might have to do in the l
Thank you Maxim and Aljoscha.
Yes the beforeEvict and afterEvict should able address point 3.
I have one more use case in my mind (which I might have to do in the later
stages of POC).
What if the `evictAfter` should behave differently based on the window
function.
For example.
I have a window t
Actually for such evictor to be useful the window should be sorted by some
field, usually event time. What do you think about adding sorted window
abstraction?
On Wed, Jul 6, 2016 at 11:36 AM, Aljoscha Krettek
wrote:
> @Maxim: That's perfect I didn't think about using Iterator.remove() for
> tha
@Maxim: That's perfect I didn't think about using Iterator.remove() for
that. I'll update the doc. What do you think Vishnu? This should also cover
your before/after case nicely.
@Vishnu: The steps would be these:
- Converge on a design in this discussion
- Add a Jira issue here: https://issues.
The new API forces iteration through every element of the buffer even if a
single value to be evicted. What about implementing Iterator.remove()
method for elements? The API would look like:
public interface Evictor extends Serializable {
/**
* Optionally evicts elements. Called before wi
Hi Aljoscha,
Thanks. Yes the new interface seems to address points 1 and 2. of
*1) I am having a use case where I have to create a custom Evictor that
will evict elements from the window based on the value (e.g., if I have
elements are of case class Item(id: Int, type:String) then evict elements
Hi,
as mentioned in the thread on improving the Windowing API I also have a
design doc just for improving WindowEvictors. I had this in my head for a
while but was hesitant to publish but since people are asking about this
now might be a good time to post it. Here's the doc:
https://docs.google.com
28 matches
Mail list logo
