Hi,
I have created a FLIP page for this enhancement
https://cwiki.apache.org/confluence/display/FLINK/FLIP-4+%3A+Enhance+Window+Evictor
Thanks,
Vishnu
On Thu, Jul 21, 2016 at 6:53 AM, Vishnu Viswanath <
vishnu.viswanat...@gmail.com> wrote:
> Thanks Aljoscha.
>
> On Thu, Jul 21, 2016 at 4:46 AM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
>> Hi,
>> this, in fact, seems to be a bug. There should be something like
>> windowState.clear();
>> for (IN element: projectedContents) {
>> windowState.add(element);
>> }
>>
>> after passing the elements to the window function.
>>
>> This is very inefficient but the only way I see of doing it right now.
>>
>> Cheers,
>> Aljoscha
>>
>>
>> On Thu, 21 Jul 2016 at 01:32 Vishnu Viswanath <
>> vishnu.viswanat...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > When we use RocksDB as state backend, how does the backend state get
>> > updated after some elements are evicted from the window?
>> > I don't see any update call being made to remove the element from the
>> state
>> > stored in RocksDB.
>> >
>> > It looks like the RocksDBListState is only having get() and add()
>> methods
>> > since it is an AppendingState, but that causes the evicted elements to
>> come
>> > back when the trigger is fired next time. (It works fine when I use
>> > MemoryStateBackend)
>> >
>> > Is this expected behavior or am I missing something.
>> >
>> > Thanks,
>> > Vishnu
>> >
>> > On Mon, Jul 18, 2016 at 7:15 AM, Vishnu Viswanath <
>> > vishnu.viswanat...@gmail.com> wrote:
>> >
>> > > Hi Aljoscha,
>> > >
>> > > Thanks! Yes, I have the create page option now in wiki.
>> > >
>> > > Regards,
>> > > Vishnu Viswanath,
>> > >
>> > > On Mon, Jul 18, 2016 at 6:34 AM, Aljoscha Krettek <
>> aljos...@apache.org>
>> > > wrote:
>> > >
>> > >> @Radu, addition of more window types and sorting should be part of
>> > another
>> > >> design proposal. This is interesting stuff but I think we should keep
>> > >> issues separated because things can get complicated very quickly.
>> > >>
>> > >> On Mon, 18 Jul 2016 at 12:32 Aljoscha Krettek <aljos...@apache.org>
>> > >> wrote:
>> > >>
>> > >> > Hi,
>> > >> > about TimeEvictor, yes, I think there should be specific evictors
>> for
>> > >> > processing time and event time. Also, the current time should be
>> > >> > retrievable from the EvictorContext.
>> > >> >
>> > >> > For the wiki you will need permissions. This was recently changed
>> > >> because
>> > >> > there was too much spam. I gave you permission to add pages. Can
>> you
>> > >> please
>> > >> > try and check if it works?
>> > >> >
>> > >> > Cheers,
>> > >> > Aljoscha
>> > >> >
>> > >> > On Fri, 15 Jul 2016 at 13:28 Vishnu Viswanath <
>> > >> > vishnu.viswanat...@gmail.com> wrote:
>> > >> >
>> > >> >> Hi all,
>> > >> >>
>> > >> >> How do we create a FLIP page, is there any permission setup
>> > required? I
>> > >> >> don't see any "Create" page(after logging in) option in the
>> header as
>> > >> >> mentioned in
>> > >> >>
>> > >> >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>> > >> >>
>> > >> >>
>> > >> >> Thanks,
>> > >> >> Vishnu
>> > >> >>
>> > >> >> On Wed, Jul 13, 2016 at 10:22 PM, Vishnu Viswanath <
>> > >> >> vishnu.viswanat...@gmail.com> wrote:
>> > >> >>
>> > >> >> > Hi Aljoscha,
>> > >> >> >
>> > >> >> > I agree, the user will know exactly that they are creating an
>> > >> EventTime
>> > >> >> > based evictor or ProcessingTime based evictor looking at the
>> code.
>> > >> >> > So do you think it will be ok to have multiple versions of
>> > >> TimeEvictor
>> > >> >> > (one for event time and one for processing time) and also a
>> > >> DeltaEvcitor
>> > >> >> > (again 2 versions- for event time and processing time) ?
>> > >> >> >
>> > >> >> > Please note that the existing behavior of
>> TimeEvictor/DeltaEvictor
>> > >> does
>> > >> >> > not consider if it is EventTime or ProcessingTime
>> > >> >> > e.g., in TimeEvictor the current time is considered as the
>> > timestamp
>> > >> of
>> > >> >> > the last element in the window
>> > >> >> >
>> > >> >> > *long currentTime = Iterables.getLast(elements).getTimestamp();*
>> > >> >> >
>> > >> >> > not the highest timestamp of all elements
>> > >> >> > what I am trying to achieve is something like:
>> > >> >> >
>> > >> >> > *long currentTime;*
>> > >> >> > * if (ctx.isEventTime()) {*
>> > >> >> > * currentTime = getMaxTimestamp(elements);*
>> > >> >> > * } else {*
>> > >> >> > * currentTime = Iterables.getLast(elements).getTimestamp();*
>> > >> >> > * }*
>> > >> >> >
>> > >> >> > Similarly, in DeltaEvictor the *`lastElement`* is
>> > >> >> > *`Iterables.getLast(elements);`* and I am thinking we should
>> > consider
>> > >> >> the
>> > >> >> > element with max timestamp as the last element instead of just
>> > >> getting
>> > >> >> the
>> > >> >> > last inserted element as *`lastElement`*
>> > >> >> >
>> > >> >> > Do you think it is the right thing to do or leave the behavior
>> > >> Evictors
>> > >> >> as
>> > >> >> > is, w.r.t to choosing the last element?
>> > >> >> >
>> > >> >> > Thanks,
>> > >> >> > Vishnu
>> > >> >> >
>> > >> >> > On Wed, Jul 13, 2016 at 11:07 AM, Aljoscha Krettek <
>> > >> aljos...@apache.org
>> > >> >> >
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> >> I still think it should be explicit in the class. For example,
>> if
>> > >> you
>> > >> >> have
>> > >> >> >> this code:
>> > >> >> >>
>> > >> >> >> input
>> > >> >> >> .keyBy()
>> > >> >> >> .window()
>> > >> >> >> .trigger(EventTimeTrigger.create())
>> > >> >> >> .evictor(TimeTrigger.create())
>> > >> >> >>
>> > >> >> >> the time behavior of the trigger is explicitly specified while
>> the
>> > >> >> evictor
>> > >> >> >> would dynamically adapt based on internal workings that the
>> user
>> > >> might
>> > >> >> not
>> > >> >> >> be aware of. Having the behavior explicit at the call site is
>> very
>> > >> >> >> important, in my opinion.
>> > >> >> >>
>> > >> >> >> On Wed, 13 Jul 2016 at 16:28 Vishnu Viswanath <
>> > >> >> >> vishnu.viswanat...@gmail.com>
>> > >> >> >> wrote:
>> > >> >> >>
>> > >> >> >> > Hi,
>> > >> >> >> >
>> > >> >> >> > I was hoping to use the isEventTime method in the
>> WindowAssigner
>> > >> to
>> > >> >> set
>> > >> >> >> > that information in the EvictorContext.
>> > >> >> >> > What do you think?.
>> > >> >> >> >
>> > >> >> >> > Thanks and Regards,
>> > >> >> >> > Vishnu Viswanath,
>> > >> >> >> >
>> > >> >> >> > On Wed, Jul 13, 2016 at 10:09 AM, Aljoscha Krettek <
>> > >> >> aljos...@apache.org
>> > >> >> >> >
>> > >> >> >> > wrote:
>> > >> >> >> >
>> > >> >> >> > > Hi,
>> > >> >> >> > > I think the way to go here is to add both an
>> EventTimeEvictor
>> > >> and a
>> > >> >> >> > > ProcessingTimeEvictor. The problem is that "isEventTime"
>> > cannot
>> > >> >> >> really be
>> > >> >> >> > > determined. That's also the reason why there is an
>> > >> EventTimeTrigger
>> > >> >> >> and a
>> > >> >> >> > > ProcessingTimeTrigger. It was just an oversight that the
>> > >> >> TimeEvictor
>> > >> >> >> does
>> > >> >> >> > > not also have these two versions.
>> > >> >> >> > >
>> > >> >> >> > > About EvictingWindowOperator, I think you can make the two
>> > >> methods
>> > >> >> >> > > non-final in WindowOperator, yes.
>> > >> >> >> > >
>> > >> >> >> > > Cheers,
>> > >> >> >> > > Aljoscha
>> > >> >> >> > >
>> > >> >> >> > > On Wed, 13 Jul 2016 at 14:32 Vishnu Viswanath <
>> > >> >> >> > > vishnu.viswanat...@gmail.com>
>> > >> >> >> > > wrote:
>> > >> >> >> > >
>> > >> >> >> > > > Hi Aljoscha,
>> > >> >> >> > > >
>> > >> >> >> > > > I am thinking of adding a method boolean isEventTime();
>> in
>> > the
>> > >> >> >> > > > EvictorContext apart from
>> > >> >> >> > > >
>> > >> >> >> > > > long getCurrentProcessingTime();
>> > >> >> >> > > > MetricGroup getMetricGroup();
>> > >> >> >> > > > long getCurrentWatermark();
>> > >> >> >> > > >
>> > >> >> >> > > > This method can be used to make the Evictor not iterate
>> > >> through
>> > >> >> all
>> > >> >> >> the
>> > >> >> >> > > > elements in TimeEvictor. There will be a few changes in
>> the
>> > >> >> existing
>> > >> >> >> > > > behavior of TimeEvictor and DeltaEvictor (I have
>> mentioned
>> > >> this
>> > >> >> in
>> > >> >> >> the
>> > >> >> >> > > > design doc)
>> > >> >> >> > > >
>> > >> >> >> > > > Also, is there any specific reason why the open and close
>> > >> method
>> > >> >> in
>> > >> >> >> > > > WindowEvictor is made final? Since the EvictorContext
>> will
>> > be
>> > >> in
>> > >> >> the
>> > >> >> >> > > > EvictingWindowOperator, I need to override the open and
>> > close
>> > >> in
>> > >> >> >> > > > EvitingWindowOperator to make the reference of
>> > EvictorContext
>> > >> >> null.
>> > >> >> >> > > >
>> > >> >> >> > > > Thanks and Regards,
>> > >> >> >> > > > Vishnu Viswanath,
>> > >> >> >> > > >
>> > >> >> >> > > > On Fri, Jul 8, 2016 at 7:40 PM, Vishnu Viswanath <
>> > >> >> >> > > > vishnu.viswanat...@gmail.com> wrote:
>> > >> >> >> > > >
>> > >> >> >> > > > My thought process when asking if we can use state
>> backend
>> > in
>> > >> >> window
>> > >> >> >> > > > > function was : can we add the elements to be evicted
>> into
>> > >> some
>> > >> >> >> state
>> > >> >> >> > > and
>> > >> >> >> > > > > allow the evictAfter to read it from some context and
>> > >> remove it
>> > >> >> >> from
>> > >> >> >> > > the
>> > >> >> >> > > > > window?
>> > >> >> >> > > > >
>> > >> >> >> > > > >
>> > >> >> >> > > > > On Fri, Jul 8, 2016 at 7:30 PM, Vishnu Viswanath <
>> > >> >> >> > > > > vishnu.viswanat...@gmail.com> wrote:
>> > >> >> >> > > > >
>> > >> >> >> > > > >> Hi Aljoscha,
>> > >> >> >> > > > >>
>> > >> >> >> > > > >> Thanks for the explanation, and sorry for late reply
>> was
>> > >> busy
>> > >> >> >> with
>> > >> >> >> > > work.
>> > >> >> >> > > > >>
>> > >> >> >> > > > >> I did think about this scenario, in fact in my
>> previous
>> > >> mail I
>> > >> >> >> > thought
>> > >> >> >> > > > of
>> > >> >> >> > > > >> posting this question, then I understood that this
>> > problem
>> > >> >> will
>> > >> >> >> be
>> > >> >> >> > > > >> there which ever method we choose(Trigger looking for
>> > >> pattern
>> > >> >> or
>> > >> >> >> > > Window
>> > >> >> >> > > > >> looking for pattern).
>> > >> >> >> > > > >>
>> > >> >> >> > > > >> I do have a pretty good watermark but my concern is
>> that
>> > it
>> > >> >> >> changes
>> > >> >> >> > > > based
>> > >> >> >> > > > >> on the key of these messages(I don't know if it is
>> > >> possible,
>> > >> >> >> haven't
>> > >> >> >> > > > >> started coding that yet. May be you could tell me).
>> Even
>> > if
>> > >> >> it is
>> > >> >> >> > yes
>> > >> >> >> > > > some
>> > >> >> >> > > > >> of these watermarks will be long(in days), which I
>> don't
>> > >> want
>> > >> >> the
>> > >> >> >> > > > trigger
>> > >> >> >> > > > >> to wait that long.
>> > >> >> >> > > > >>
>> > >> >> >> > > > >> It looks like it is not easy to have an evictAfter
>> based
>> > on
>> > >> >> >> window
>> > >> >> >> > > > >> function(without introducing coupling), but can the
>> > current
>> > >> >> >> window
>> > >> >> >> > > apply
>> > >> >> >> > > > >> function be modified to allow it to change the
>> elements
>> > in
>> > >> it
>> > >> >> -
>> > >> >> >> may
>> > >> >> >> > be
>> > >> >> >> > > > >> using some state backend(I don't know how excatly the
>> > >> >> internals
>> > >> >> >> of
>> > >> >> >> > > these
>> > >> >> >> > > > >> work, so this might be a wrong question)
>> > >> >> >> > > > >>
>> > >> >> >> > > > >> Thanks and Regards,
>> > >> >> >> > > > >> Vishnu Viswanath,
>> > >> >> >> > > > >>
>> > >> >> >> > > > >> On Fri, Jul 8, 2016 at 10:20 AM, Aljoscha Krettek <
>> > >> >> >> > > aljos...@apache.org>
>> > >> >> >> > > > >> wrote:
>> > >> >> >> > > > >>
>> > >> >> >> > > > >>> Hi Vishnu,
>> > >> >> >> > > > >>> how long would these patterns be? The Trigger would
>> not
>> > >> have
>> > >> >> to
>> > >> >> >> > sort
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> elements for every new element but just insert the
>> new
>> > >> >> element
>> > >> >> >> into
>> > >> >> >> > > an
>> > >> >> >> > > > >>> internal data structure. Only when it sees that the
>> > >> >> watermark is
>> > >> >> >> > > past a
>> > >> >> >> > > > >>> certain point would it check whether the pattern
>> matches
>> > >> and
>> > >> >> >> > actually
>> > >> >> >> > > > >>> Trigger.
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> A general note regarding order and event time: I
>> think
>> > >> >> relying
>> > >> >> >> on
>> > >> >> >> > > this
>> > >> >> >> > > > >>> for
>> > >> >> >> > > > >>> computation is very tricky unless the watermark is
>> 100 %
>> > >> >> >> correct or
>> > >> >> >> > > you
>> > >> >> >> > > > >>> completely discard elements that arrive after the
>> > >> watermark,
>> > >> >> >> i.e.
>> > >> >> >> > > > >>> elements
>> > >> >> >> > > > >>> that would break the promise of the watermark that no
>> > >> >> elements
>> > >> >> >> with
>> > >> >> >> > > an
>> > >> >> >> > > > >>> earlier timestamp will ever arrive. The reason for
>> this
>> > is
>> > >> >> that
>> > >> >> >> > there
>> > >> >> >> > > > >>> could
>> > >> >> >> > > > >>> always enter new elements that end up between already
>> > seen
>> > >> >> >> > elements.
>> > >> >> >> > > > For
>> > >> >> >> > > > >>> example, let's say we have this sequence of elements
>> > when
>> > >> the
>> > >> >> >> > trigger
>> > >> >> >> > > > >>> fires:
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> a-b-a
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> This is the sequence that you are looking for and you
>> > emit
>> > >> >> some
>> > >> >> >> > > result
>> > >> >> >> > > > >>> from
>> > >> >> >> > > > >>> the WindowFunction. Now, new elements arrive that
>> fall
>> > in
>> > >> >> >> between
>> > >> >> >> > the
>> > >> >> >> > > > >>> elements we already have:
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> a-d-e-b-f-g-a
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> This is an updated, sorted view of the actual
>> event-time
>> > >> >> stream
>> > >> >> >> and
>> > >> >> >> > > we
>> > >> >> >> > > > >>> didn't realize that the stream actually looks like
>> this
>> > >> >> before.
>> > >> >> >> > Does
>> > >> >> >> > > > this
>> > >> >> >> > > > >>> still match the original pattern or should we now
>> > consider
>> > >> >> this
>> > >> >> >> as
>> > >> >> >> > > > >>> non-matching? If no, then the earlier successful
>> match
>> > for
>> > >> >> a-b-a
>> > >> >> >> > was
>> > >> >> >> > > > >>> wrong
>> > >> >> >> > > > >>> and we should never have processed it but we didn't
>> know
>> > >> at
>> > >> >> the
>> > >> >> >> > time.
>> > >> >> >> > > > If
>> > >> >> >> > > > >>> yes, then pattern matching like this can be done in
>> the
>> > >> >> Trigger
>> > >> >> >> by
>> > >> >> >> > > > having
>> > >> >> >> > > > >>> something like pattern slots: You don't have to store
>> > all
>> > >> >> >> elements
>> > >> >> >> > in
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> Trigger, you just need to store possible candidates
>> that
>> > >> >> could
>> > >> >> >> > match
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> pattern and ignore the other (in-between) elements.
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> Cheers,
>> > >> >> >> > > > >>> Aljoscha
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> On Fri, 8 Jul 2016 at 14:10 Vishnu Viswanath <
>> > >> >> >> > > > >>> vishnu.viswanat...@gmail.com>
>> > >> >> >> > > > >>> wrote:
>> > >> >> >> > > > >>>
>> > >> >> >> > > > >>> > Hi Aljoscha,
>> > >> >> >> > > > >>> >
>> > >> >> >> > > > >>> > That is a good idea, trying to tie it back to the
>> use
>> > >> case,
>> > >> >> >> > > > >>> > e.g., suppose trigger is looking for a pattern,
>> a-b-a
>> > >> and
>> > >> >> >> when it
>> > >> >> >> > > > sees
>> > >> >> >> > > > >>> such
>> > >> >> >> > > > >>> > a pattern, it will trigger the window and it knows
>> > that
>> > >> now
>> > >> >> >> the
>> > >> >> >> > > > >>> Evictor is
>> > >> >> >> > > > >>> > going to evict the element b, and trigger updates
>> its
>> > >> >> state as
>> > >> >> >> > a-a
>> > >> >> >> > > > >>> (even
>> > >> >> >> > > > >>> > before the window & evictor completes) and will be
>> > >> looking
>> > >> >> for
>> > >> >> >> > the
>> > >> >> >> > > > >>> rest of
>> > >> >> >> > > > >>> > the pattern i.e., b-a. But I can think of 1 problem
>> > >> here,
>> > >> >> >> > > > >>> >
>> > >> >> >> > > > >>> > - the events can arrive out of order, i.e., the
>> > >> trigger
>> > >> >> >> might
>> > >> >> >> > be
>> > >> >> >> > > > >>> seeing
>> > >> >> >> > > > >>> > a pattern a-a-b but actual event time is a-b-a
>> then
>> > >> >> trigger
>> > >> >> >> > will
>> > >> >> >> > > > >>> have to
>> > >> >> >> > > > >>> > sort the elements in the window everytime it
>> sees
>> > an
>> > >> >> >> element.
>> > >> >> >> > (I
>> > >> >> >> > > > was
>> > >> >> >> > > > >>> > planning to do this sorting in the window, which
>> > >> will be
>> > >> >> >> less
>> > >> >> >> > > > often
>> > >> >> >> > > > >>> -
>> > >> >> >> > > > >>> > only
>> > >> >> >> > > > >>> > when the trigger fires)
>> > >> >> >> > > > >>> >
>> > >> >> >> > > > >>> > Thanks and Regards,
>> > >> >> >> > > > >>> > Vishnu Viswanath,
>> > >> >> >> > > > >>> >
>> > >> >> >> > > > >>> > On Fri, Jul 8, 2016 at 6:04 AM, Aljoscha Krettek <
>> > >> >> >> > > > aljos...@apache.org>
>> > >> >> >> > > > >>> > wrote:
>> > >> >> >> > > > >>> >
>> > >> >> >> > > > >>> > Hi,
>> > >> >> >> > > > >>> > > come to think of it, the right place to put such
>> > >> checks
>> > >> >> is
>> > >> >> >> > > actually
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > Trigger. It would have to be a custom trigger
>> that
>> > >> >> observes
>> > >> >> >> > time
>> > >> >> >> > > > but
>> > >> >> >> > > > >>> also
>> > >> >> >> > > > >>> > > keeps some internal state machine to decide when
>> it
>> > >> has
>> > >> >> >> > observed
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> > right
>> > >> >> >> > > > >>> > > pattern in the window. Then the window function
>> > would
>> > >> >> just
>> > >> >> >> have
>> > >> >> >> > > to
>> > >> >> >> > > > >>> do the
>> > >> >> >> > > > >>> > > processing and you have good separation of
>> concerns.
>> > >> Does
>> > >> >> >> that
>> > >> >> >> > > make
>> > >> >> >> > > > >>> > sense?
>> > >> >> >> > > > >>> > >
>> > >> >> >> > > > >>> > > I'm ignoring time and sorting by time for now
>> > because
>> > >> we
>> > >> >> >> > probably
>> > >> >> >> > > > >>> need
>> > >> >> >> > > > >>> > > another design document for that. To me it seems
>> > like
>> > >> a
>> > >> >> >> bigger
>> > >> >> >> > > > thing.
>> > >> >> >> > > > >>> > >
>> > >> >> >> > > > >>> > > Cheers,
>> > >> >> >> > > > >>> > > Aljoscha
>> > >> >> >> > > > >>> > >
>> > >> >> >> > > > >>> > > On Thu, 7 Jul 2016 at 23:56 Vishnu Viswanath <
>> > >> >> >> > > > >>> > vishnu.viswanat...@gmail.com
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > wrote:
>> > >> >> >> > > > >>> > >
>> > >> >> >> > > > >>> > > > Hi,
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > > Regarding the evictAfter function, that evicts
>> > >> based on
>> > >> >> >> some
>> > >> >> >> > > > >>> decision
>> > >> >> >> > > > >>> > > made
>> > >> >> >> > > > >>> > > > by the window function: I think it will be
>> nice
>> > if
>> > >> we
>> > >> >> can
>> > >> >> >> > come
>> > >> >> >> > > > up
>> > >> >> >> > > > >>> with
>> > >> >> >> > > > >>> > > > something that is LESS coupled, because I can
>> > think
>> > >> of
>> > >> >> >> > several
>> > >> >> >> > > > use
>> > >> >> >> > > > >>> > cases
>> > >> >> >> > > > >>> > > > that depend on it.
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > > Especially in the case where there are late
>> > arriving
>> > >> >> >> > messages.
>> > >> >> >> > > > Only
>> > >> >> >> > > > >>> > after
>> > >> >> >> > > > >>> > > > the window function is applied we could tell
>> what
>> > >> to do
>> > >> >> >> with
>> > >> >> >> > > the
>> > >> >> >> > > > >>> > elements
>> > >> >> >> > > > >>> > > > in the window. You could apply your business
>> logic
>> > >> >> there
>> > >> >> >> to
>> > >> >> >> > > > >>> determine
>> > >> >> >> > > > >>> > if
>> > >> >> >> > > > >>> > > > the window funciton was able to do what it is
>> > >> supposed
>> > >> >> to
>> > >> >> >> do,
>> > >> >> >> > > if
>> > >> >> >> > > > >>> yes
>> > >> >> >> > > > >>> > > evict
>> > >> >> >> > > > >>> > > > those elements, else(since the elements you are
>> > >> looking
>> > >> >> >> for
>> > >> >> >> > > > haven't
>> > >> >> >> > > > >>> > > arrived
>> > >> >> >> > > > >>> > > > yet) wait and try again when the trigger gets
>> > fired
>> > >> >> next
>> > >> >> >> > time.
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > > Thanks and Regards,
>> > >> >> >> > > > >>> > > > Vishnu Viswanath,
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > > On Thu, Jul 7, 2016 at 9:19 AM, Radu Tudoran <
>> > >> >> >> > > > >>> radu.tudo...@huawei.com>
>> > >> >> >> > > > >>> > > > wrote:
>> > >> >> >> > > > >>> > > >
>> > >> >> >> > > > >>> > > > > Hi,
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > @Aljoscha - I can understand the reason why
>> you
>> > >> are
>> > >> >> >> > hesitant
>> > >> >> >> > > to
>> > >> >> >> > > > >>> > > introduce
>> > >> >> >> > > > >>> > > > > "slower" windows such as the ones that would
>> > >> maintain
>> > >> >> >> > sorted
>> > >> >> >> > > > >>> items or
>> > >> >> >> > > > >>> > > > > windows with bindings between the different
>> > >> entities
>> > >> >> >> > > (evictor,
>> > >> >> >> > > > >>> > trigger,
>> > >> >> >> > > > >>> > > > > window, apply function). However, I think
>> it's
>> > >> >> possible
>> > >> >> >> > just
>> > >> >> >> > > to
>> > >> >> >> > > > >>> > create
>> > >> >> >> > > > >>> > > > more
>> > >> >> >> > > > >>> > > > > types of windows. The existing ones
>> > (timewindows,
>> > >> >> global
>> > >> >> >> > > > windows
>> > >> >> >> > > > >>> ...)
>> > >> >> >> > > > >>> > > can
>> > >> >> >> > > > >>> > > > > remain, and just add some more flavors of
>> > windows
>> > >> >> were
>> > >> >> >> more
>> > >> >> >> > > > >>> features
>> > >> >> >> > > > >>> > > are
>> > >> >> >> > > > >>> > > > > enabled or more functionality (e.g., access
>> to
>> > the
>> > >> >> each
>> > >> >> >> > > element
>> > >> >> >> > > > >>> in
>> > >> >> >> > > > >>> > the
>> > >> >> >> > > > >>> > > > > evictor ; possibility to delete or mark for
>> > >> eviction
>> > >> >> >> > elements
>> > >> >> >> > > > in
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > > > function...)
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > Regarding the specific case of sorted
>> windows, I
>> > >> >> think
>> > >> >> >> the
>> > >> >> >> > N
>> > >> >> >> > > > lon
>> > >> >> >> > > > >>> N
>> > >> >> >> > > > >>> > > > > complexity to sort (the worst case) is very
>> > >> >> unlikely. In
>> > >> >> >> > fact
>> > >> >> >> > > > you
>> > >> >> >> > > > >>> > have
>> > >> >> >> > > > >>> > > > > almost sorted items/arrays. Moreover, if you
>> > >> consider
>> > >> >> >> that
>> > >> >> >> > in
>> > >> >> >> > > > >>> > > iteration X
>> > >> >> >> > > > >>> > > > > all elements were sorted, then in iteration
>> X+1
>> > >> you
>> > >> >> will
>> > >> >> >> > need
>> > >> >> >> > > > to
>> > >> >> >> > > > >>> sort
>> > >> >> >> > > > >>> > > > just
>> > >> >> >> > > > >>> > > > > the newly arrived elements (M). I would
>> expect
>> > >> that
>> > >> >> this
>> > >> >> >> > > > number M
>> > >> >> >> > > > >>> > might
>> > >> >> >> > > > >>> > > > be
>> > >> >> >> > > > >>> > > > > significant smaller then N (elements that
>> > exists).
>> > >> >> Then
>> > >> >> >> > using
>> > >> >> >> > > > an
>> > >> >> >> > > > >>> > > > insertion
>> > >> >> >> > > > >>> > > > > sort for these new elements you would have
>> M *
>> > N
>> > >> >> >> > complexity
>> > >> >> >> > > > and
>> > >> >> >> > > > >>> if
>> > >> >> >> > > > >>> > > M<< N
>> > >> >> >> > > > >>> > > > > then the complexity is O(N). Alternatively
>> you
>> > can
>> > >> >> use a
>> > >> >> >> > > binary
>> > >> >> >> > > > >>> > search
>> > >> >> >> > > > >>> > > > for
>> > >> >> >> > > > >>> > > > > insertion and then you further reduce the
>> > >> complexity
>> > >> >> to
>> > >> >> >> > > > O(logN).
>> > >> >> >> > > > >>> > > > > If M is proportional to N then you can sort M
>> > and
>> > >> use
>> > >> >> >> merge
>> > >> >> >> > > > sort
>> > >> >> >> > > > >>> for
>> > >> >> >> > > > >>> > > > > combining.
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > Dr. Radu Tudoran
>> > >> >> >> > > > >>> > > > > Research Engineer - Big Data Expert
>> > >> >> >> > > > >>> > > > > IT R&D Division
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
>> > >> >> >> > > > >>> > > > > European Research Center
>> > >> >> >> > > > >>> > > > > Riesstrasse 25, 80992 München
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > E-mail: radu.tudo...@huawei.com
>> > >> >> >> > > > >>> > > > > Mobile: +49 15209084330
>> > >> >> >> > > > >>> > > > > Telephone: +49 891588344173
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
>> > >> >> >> > > > >>> > > > > Hansaallee 205, 40549 Düsseldorf, Germany,
>> > >> >> >> www.huawei.com
>> > >> >> >> > > > >>> > > > > Registered Office: Düsseldorf, Register Court
>> > >> >> >> Düsseldorf,
>> > >> >> >> > HRB
>> > >> >> >> > > > >>> 56063,
>> > >> >> >> > > > >>> > > > > Managing Director: Bo PENG, Wanzhou MENG,
>> Lifang
>> > >> CHEN
>> > >> >> >> > > > >>> > > > > Sitz der Gesellschaft: Düsseldorf,
>> Amtsgericht
>> > >> >> >> Düsseldorf,
>> > >> >> >> > > HRB
>> > >> >> >> > > > >>> 56063,
>> > >> >> >> > > > >>> > > > > Geschäftsführer: Bo PENG, Wanzhou MENG,
>> Lifang
>> > >> CHEN
>> > >> >> >> > > > >>> > > > > This e-mail and its attachments contain
>> > >> confidential
>> > >> >> >> > > > information
>> > >> >> >> > > > >>> from
>> > >> >> >> > > > >>> > > > > HUAWEI, which is intended only for the
>> person or
>> > >> >> entity
>> > >> >> >> > whose
>> > >> >> >> > > > >>> address
>> > >> >> >> > > > >>> > > is
>> > >> >> >> > > > >>> > > > > listed above. Any use of the information
>> > contained
>> > >> >> >> herein
>> > >> >> >> > in
>> > >> >> >> > > > any
>> > >> >> >> > > > >>> way
>> > >> >> >> > > > >>> > > > > (including, but not limited to, total or
>> partial
>> > >> >> >> > disclosure,
>> > >> >> >> > > > >>> > > > reproduction,
>> > >> >> >> > > > >>> > > > > or dissemination) by persons other than the
>> > >> intended
>> > >> >> >> > > > >>> recipient(s) is
>> > >> >> >> > > > >>> > > > > prohibited. If you receive this e-mail in
>> error,
>> > >> >> please
>> > >> >> >> > > notify
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > sender
>> > >> >> >> > > > >>> > > > > by phone or email immediately and delete it!
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > -----Original Message-----
>> > >> >> >> > > > >>> > > > > From: 吕文龙(吕文龙) [mailto:
>> > >> wenlong....@alibaba-inc.com]
>> > >> >> >> > > > >>> > > > > Sent: Thursday, July 07, 2016 11:59 AM
>> > >> >> >> > > > >>> > > > > To: dev@flink.apache.org
>> > >> >> >> > > > >>> > > > > Subject: 答复: [DISCUSS] Enhance Window
>> Evictor in
>> > >> >> Flink
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > HI,
>> > >> >> >> > > > >>> > > > > I think it is necessary to support sorted
>> > window,
>> > >> >> which
>> > >> >> >> can
>> > >> >> >> > > > avoid
>> > >> >> >> > > > >>> > > > scanning
>> > >> >> >> > > > >>> > > > > all the elements of window while trying to
>> > >> evicting
>> > >> >> >> > element,
>> > >> >> >> > > > >>> which
>> > >> >> >> > > > >>> > may
>> > >> >> >> > > > >>> > > > cost
>> > >> >> >> > > > >>> > > > > many IO operations, such as querying DBs to
>> get
>> > >> >> elements
>> > >> >> >> > from
>> > >> >> >> > > > >>> state.
>> > >> >> >> > > > >>> > > > > What's more, when an window aggregation
>> function
>> > >> is
>> > >> >> >> > > invertible,
>> > >> >> >> > > > >>> such
>> > >> >> >> > > > >>> > as
>> > >> >> >> > > > >>> > > > > sum, which can be updated by adding or
>> removing
>> > a
>> > >> >> single
>> > >> >> >> > > > record,
>> > >> >> >> > > > >>> > window
>> > >> >> >> > > > >>> > > > > results can be incrementally calculated. In
>> this
>> > >> >> kind of
>> > >> >> >> > > case,
>> > >> >> >> > > > >>> we can
>> > >> >> >> > > > >>> > > > > dramatically improve the performance of
>> window
>> > >> >> >> aggregation,
>> > >> >> >> > > if
>> > >> >> >> > > > >>> > evictor
>> > >> >> >> > > > >>> > > > can
>> > >> >> >> > > > >>> > > > > trigger update of window aggregation state by
>> > some
>> > >> >> >> > mechanism.
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > Best Wishes!
>> > >> >> >> > > > >>> > > > > ---
>> > >> >> >> > > > >>> > > > > wenlong.lwl
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > -----邮件原件-----
>> > >> >> >> > > > >>> > > > > 发件人: Aljoscha Krettek [mailto:
>> > aljos...@apache.org
>> > >> ]
>> > >> >> >> > > > >>> > > > > 发送时间: 2016年7月7日 17:32
>> > >> >> >> > > > >>> > > > > 收件人: dev@flink.apache.org
>> > >> >> >> > > > >>> > > > > 主题: Re: [DISCUSS] Enhance Window Evictor in
>> > Flink
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > Hi,
>> > >> >> >> > > > >>> > > > > regarding "sorting the window by event
>> time": I
>> > >> also
>> > >> >> >> > > considered
>> > >> >> >> > > > >>> this
>> > >> >> >> > > > >>> > > but
>> > >> >> >> > > > >>> > > > > in the end I don't think it's necessary.
>> Sorting
>> > >> is
>> > >> >> >> rather
>> > >> >> >> > > > >>> expensive
>> > >> >> >> > > > >>> > > and
>> > >> >> >> > > > >>> > > > > making decisions based on the order of
>> elements
>> > >> can
>> > >> >> be
>> > >> >> >> > > tricky.
>> > >> >> >> > > > An
>> > >> >> >> > > > >>> > > extreme
>> > >> >> >> > > > >>> > > > > example of why this can be problematic is the
>> > case
>> > >> >> where
>> > >> >> >> > all
>> > >> >> >> > > > >>> elements
>> > >> >> >> > > > >>> > > in
>> > >> >> >> > > > >>> > > > > the window have the same timestamp. Now, if
>> you
>> > >> >> decide
>> > >> >> >> to
>> > >> >> >> > > evict
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > > first 5
>> > >> >> >> > > > >>> > > > > elements based on timestamp order you
>> basically
>> > >> >> >> arbitrarily
>> > >> >> >> > > > >>> evict 5
>> > >> >> >> > > > >>> > > > > elements. I think the better solution for
>> doing
>> > >> >> >> time-based
>> > >> >> >> > > > >>> eviction
>> > >> >> >> > > > >>> > is
>> > >> >> >> > > > >>> > > to
>> > >> >> >> > > > >>> > > > > do one pass over the elements to get an
>> overview
>> > >> of
>> > >> >> the
>> > >> >> >> > > > timestamp
>> > >> >> >> > > > >>> > > > > distribution, then do a second pass and evict
>> > >> >> elements
>> > >> >> >> > based
>> > >> >> >> > > on
>> > >> >> >> > > > >>> what
>> > >> >> >> > > > >>> > > was
>> > >> >> >> > > > >>> > > > > learned in the first pass. This has
>> complexity
>> > 2*n
>> > >> >> >> compared
>> > >> >> >> > > to
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > n*log
>> > >> >> >> > > > >>> > > > n
>> > >> >> >> > > > >>> > > > > (plus the work of actually deciding what to
>> > >> evict) of
>> > >> >> >> the
>> > >> >> >> > > sort
>> > >> >> >> > > > >>> based
>> > >> >> >> > > > >>> > > > > strategy.
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > I might be wrong, though, and there could be
>> a
>> > >> valid
>> > >> >> >> > use-case
>> > >> >> >> > > > not
>> > >> >> >> > > > >>> > > covered
>> > >> >> >> > > > >>> > > > > by the above idea.
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > regarding Vishnu's other use case of evicting
>> > >> based
>> > >> >> on
>> > >> >> >> some
>> > >> >> >> > > > >>> decision
>> > >> >> >> > > > >>> > in
>> > >> >> >> > > > >>> > > > the
>> > >> >> >> > > > >>> > > > > WindowFunction: could this be solved by doing
>> > the
>> > >> >> check
>> > >> >> >> for
>> > >> >> >> > > the
>> > >> >> >> > > > >>> > pattern
>> > >> >> >> > > > >>> > > > in
>> > >> >> >> > > > >>> > > > > the evictor itself instead of in the window
>> > >> function?
>> > >> >> >> I'm
>> > >> >> >> > > very
>> > >> >> >> > > > >>> > hesitant
>> > >> >> >> > > > >>> > > > to
>> > >> >> >> > > > >>> > > > > introduce a coupling between the different
>> > >> >> components of
>> > >> >> >> > the
>> > >> >> >> > > > >>> > windowing
>> > >> >> >> > > > >>> > > > > system, i.e. assigner, trigger, evictor and
>> > window
>> > >> >> >> > function.
>> > >> >> >> > > > The
>> > >> >> >> > > > >>> > reason
>> > >> >> >> > > > >>> > > > is
>> > >> >> >> > > > >>> > > > > that using an evictor has a huge performance
>> > >> impact
>> > >> >> >> since
>> > >> >> >> > the
>> > >> >> >> > > > >>> system
>> > >> >> >> > > > >>> > > > always
>> > >> >> >> > > > >>> > > > > has to keep all elements and cannot to
>> > incremental
>> > >> >> >> > > aggregation
>> > >> >> >> > > > of
>> > >> >> >> > > > >>> > > window
>> > >> >> >> > > > >>> > > > > results and I therefore don't want to put
>> > specific
>> > >> >> >> features
>> > >> >> >> > > > >>> regarding
>> > >> >> >> > > > >>> > > > > eviction into the other components.
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > Cheers,
>> > >> >> >> > > > >>> > > > > Aljoscha
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > On Thu, 7 Jul 2016 at 10:00 Radu Tudoran <
>> > >> >> >> > > > >>> radu.tudo...@huawei.com>
>> > >> >> >> > > > >>> > > > wrote:
>> > >> >> >> > > > >>> > > > >
>> > >> >> >> > > > >>> > > > > > Hi,
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > I think the situation Vishnu raised is
>> > something
>> > >> >> that
>> > >> >> >> > > should
>> > >> >> >> > > > be
>> > >> >> >> > > > >>> > > > > accounted.
>> > >> >> >> > > > >>> > > > > > It can happen indeed that you want to
>> > condition
>> > >> >> what
>> > >> >> >> you
>> > >> >> >> > > > evict
>> > >> >> >> > > > >>> from
>> > >> >> >> > > > >>> > > > > > the window based on the result of the
>> function
>> > >> to
>> > >> >> be
>> > >> >> >> > > applied.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > My 2 cents...
>> > >> >> >> > > > >>> > > > > > I would suggest adding a list for the
>> elements
>> > >> of
>> > >> >> the
>> > >> >> >> > > stream
>> > >> >> >> > > > >>> where
>> > >> >> >> > > > >>> > > you
>> > >> >> >> > > > >>> > > > > > can MARK them to be delete. Alternatively
>> the
>> > >> >> iterator
>> > >> >> >> > can
>> > >> >> >> > > be
>> > >> >> >> > > > >>> > > extended
>> > >> >> >> > > > >>> > > > > > to have a function
>> > >> Iterator.markForEviction(int);
>> > >> >> >> These
>> > >> >> >> > can
>> > >> >> >> > > > be
>> > >> >> >> > > > >>> made
>> > >> >> >> > > > >>> > > > > > available also in the apply function.
>> > Moreover,
>> > >> we
>> > >> >> can
>> > >> >> >> > use
>> > >> >> >> > > > >>> this to
>> > >> >> >> > > > >>> > > > > > extend the functionality such that you add
>> > MARKs
>> > >> >> >> either
>> > >> >> >> > for
>> > >> >> >> > > > >>> > eviction
>> > >> >> >> > > > >>> > > > > > after the function has finished triggering
>> or
>> > >> to be
>> > >> >> >> > evicted
>> > >> >> >> > > > in
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > next
>> > >> >> >> > > > >>> > > > > iteration.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > Dr. Radu Tudoran
>> > >> >> >> > > > >>> > > > > > Research Engineer - Big Data Expert
>> > >> >> >> > > > >>> > > > > > IT R&D Division
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
>> > >> >> >> > > > >>> > > > > > European Research Center
>> > >> >> >> > > > >>> > > > > > Riesstrasse 25, 80992 München
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > E-mail: radu.tudo...@huawei.com
>> > >> >> >> > > > >>> > > > > > Mobile: +49 15209084330
>> > >> >> >> > > > >>> > > > > > Telephone: +49 891588344173
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > HUAWEI TECHNOLOGIES Duesseldorf GmbH
>> > >> >> >> > > > >>> > > > > > Hansaallee 205, 40549 Düsseldorf, Germany,
>> > >> >> >> > www.huawei.com
>> > >> >> >> > > > >>> > Registered
>> > >> >> >> > > > >>> > > > > > Office: Düsseldorf, Register Court
>> Düsseldorf,
>> > >> HRB
>> > >> >> >> 56063,
>> > >> >> >> > > > >>> Managing
>> > >> >> >> > > > >>> > > > > > Director: Bo PENG, Wanzhou MENG, Lifang
>> CHEN
>> > >> Sitz
>> > >> >> der
>> > >> >> >> > > > >>> Gesellschaft:
>> > >> >> >> > > > >>> > > > > > Düsseldorf, Amtsgericht Düsseldorf, HRB
>> 56063,
>> > >> >> >> > > > >>> > > > > > Geschäftsführer: Bo PENG, Wanzhou MENG,
>> Lifang
>> > >> CHEN
>> > >> >> >> This
>> > >> >> >> > > > >>> e-mail and
>> > >> >> >> > > > >>> > > > > > its attachments contain confidential
>> > information
>> > >> >> from
>> > >> >> >> > > HUAWEI,
>> > >> >> >> > > > >>> which
>> > >> >> >> > > > >>> > > is
>> > >> >> >> > > > >>> > > > > > intended only for the person or entity
>> whose
>> > >> >> address
>> > >> >> >> is
>> > >> >> >> > > > listed
>> > >> >> >> > > > >>> > above.
>> > >> >> >> > > > >>> > > > > > Any use of the information contained
>> herein in
>> > >> any
>> > >> >> way
>> > >> >> >> > > > >>> (including,
>> > >> >> >> > > > >>> > > but
>> > >> >> >> > > > >>> > > > > > not limited to, total or partial
>> disclosure,
>> > >> >> >> > reproduction,
>> > >> >> >> > > or
>> > >> >> >> > > > >>> > > > > > dissemination) by persons other than the
>> > >> intended
>> > >> >> >> > > > recipient(s)
>> > >> >> >> > > > >>> is
>> > >> >> >> > > > >>> > > > > > prohibited. If you receive this e-mail in
>> > error,
>> > >> >> >> please
>> > >> >> >> > > > notify
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > > > > > sender by phone or email immediately and
>> > delete
>> > >> it!
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > -----Original Message-----
>> > >> >> >> > > > >>> > > > > > From: Vishnu Viswanath [mailto:
>> > >> >> >> > > vishnu.viswanat...@gmail.com]
>> > >> >> >> > > > >>> > > > > > Sent: Thursday, July 07, 2016 1:28 AM
>> > >> >> >> > > > >>> > > > > > To: Dev
>> > >> >> >> > > > >>> > > > > > Subject: Re: [DISCUSS] Enhance Window
>> Evictor
>> > in
>> > >> >> Flink
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > Thank you Maxim and Aljoscha.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > Yes the beforeEvict and afterEvict should
>> able
>> > >> >> address
>> > >> >> >> > > point
>> > >> >> >> > > > 3.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > I have one more use case in my mind (which
>> I
>> > >> might
>> > >> >> >> have
>> > >> >> >> > to
>> > >> >> >> > > do
>> > >> >> >> > > > >>> in
>> > >> >> >> > > > >>> > the
>> > >> >> >> > > > >>> > > > > > later stages of POC).
>> > >> >> >> > > > >>> > > > > > What if the `evictAfter` should behave
>> > >> differently
>> > >> >> >> based
>> > >> >> >> > on
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> > > window
>> > >> >> >> > > > >>> > > > > > function.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > For example.
>> > >> >> >> > > > >>> > > > > > I have a window that got triggered and my
>> > evict
>> > >> >> >> function
>> > >> >> >> > is
>> > >> >> >> > > > >>> being
>> > >> >> >> > > > >>> > > > > > called after the apply function. In such
>> > cases I
>> > >> >> >> should
>> > >> >> >> > be
>> > >> >> >> > > > >>> able to
>> > >> >> >> > > > >>> > > > > > decide on what I should evict based on the
>> > >> window
>> > >> >> >> > function.
>> > >> >> >> > > > >>> > > > > > e.g.,
>> > >> >> >> > > > >>> > > > > > let the window have elements of type `case
>> > class
>> > >> >> >> Item(id:
>> > >> >> >> > > > >>> String,
>> > >> >> >> > > > >>> > > type:
>> > >> >> >> > > > >>> > > > > > String)` and let the types be `type1` and
>> > >> `type2`.
>> > >> >> >> > > > >>> > > > > > If window function is able to find a
>> sequence
>> > :
>> > >> >> `type1
>> > >> >> >> > > type2
>> > >> >> >> > > > >>> > type1`,
>> > >> >> >> > > > >>> > > > > > then evict all elements of the type type2.
>> > >> >> >> > > > >>> > > > > > or if the window function is able to find a
>> > >> >> sequence
>> > >> >> >> > `type2
>> > >> >> >> > > > >>> type2
>> > >> >> >> > > > >>> > > > > > type1`, then evict all elements of type
>> type1
>> > >> else
>> > >> >> >> don't
>> > >> >> >> > > > evict
>> > >> >> >> > > > >>> any
>> > >> >> >> > > > >>> > > > > elements.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > Is this possible? or at least let the
>> window
>> > >> >> function
>> > >> >> >> > > choose
>> > >> >> >> > > > >>> > between
>> > >> >> >> > > > >>> > > > > > two Evictor functions -(one for success
>> case
>> > and
>> > >> >> one
>> > >> >> >> > > failure
>> > >> >> >> > > > >>> case)
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > @Maxim:
>> > >> >> >> > > > >>> > > > > > regarding the sorted window, actually I
>> wanted
>> > >> my
>> > >> >> >> > elements
>> > >> >> >> > > to
>> > >> >> >> > > > >>> be
>> > >> >> >> > > > >>> > > > > > sorted but not for the eviction but while
>> > >> applying
>> > >> >> the
>> > >> >> >> > > window
>> > >> >> >> > > > >>> > > function
>> > >> >> >> > > > >>> > > > > > (so thought this could be done easily).
>> But it
>> > >> >> would
>> > >> >> >> be
>> > >> >> >> > > good
>> > >> >> >> > > > to
>> > >> >> >> > > > >>> > have
>> > >> >> >> > > > >>> > > > > > the window sorted based on EventTime.
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > Thanks and Regards,
>> > >> >> >> > > > >>> > > > > > Vishnu Viswanath,
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > On Wed, Jul 6, 2016 at 3:55 PM, Maxim <
>> > >> >> >> mfat...@gmail.com
>> > >> >> >> > >
>> > >> >> >> > > > >>> wrote:
>> > >> >> >> > > > >>> > > > > >
>> > >> >> >> > > > >>> > > > > > > Actually for such evictor to be useful
>> the
>> > >> window
>> > >> >> >> > should
>> > >> >> >> > > be
>> > >> >> >> > > > >>> > sorted
>> > >> >> >> > > > >>> > > > > > > by some field, usually event time. What
>> do
>> > you
>> > >> >> think
>> > >> >> >> > > about
>> > >> >> >> > > > >>> adding
>> > >> >> >> > > > >>> > > > > > > sorted window abstraction?
>> > >> >> >> > > > >>> > > > > > >
>> > >> >> >> > > > >>> > > > > > > On Wed, Jul 6, 2016 at 11:36 AM, Aljoscha
>> > >> Krettek
>> > >> >> >> > > > >>> > > > > > > <aljos...@apache.org>
>> > >> >> >> > > > >>> > > > > > > wrote:
>> > >> >> >> > > > >>> > > > > > >
>> > >> >> >> > > > >>> > > > > > > > @Maxim: That's perfect I didn't think
>> > about
>> > >> >> using
>> > >> >> >> > > > >>> > > > > > > > Iterator.remove() for that. I'll update
>> > the
>> > >> >> doc.
>> > >> >> >> What
>> > >> >> >> > > do
>> > >> >> >> > > > >>> you
>> > >> >> >> > > > >>> > > think
>> > >> >> >> > > > >>> > > > > > > > Vishnu? This should also
>> > >> >> >> > > > >>> > > > > > > cover
>> > >> >> >> > > > >>> > > > > > > > your before/after case nicely.
>> > >> >> >> > > > >>> > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > @Vishnu: The steps would be these:
>> > >> >> >> > > > >>> > > > > > > > - Converge on a design in this
>> discussion
>> > >> >> >> > > > >>> > > > > > > > - Add a Jira issue here:
>> > >> >> >> > > > >>> > > > > > > >
>> > https://issues.apache.org/jira/browse/FLINK
>> > >> >> >> > > > >>> > > > > > > > - Work on the code an create a pull
>> > >> request on
>> > >> >> >> > github
>> > >> >> >> > > > >>> > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > The steps are also outlined here
>> > >> >> >> > > > >>> > > > > > > >
>> > >> http://flink.apache.org/how-to-contribute.html
>> > >> >> >> and
>> > >> >> >> > > here
>> > >> >> >> > > > >>> > > > > > > >
>> > >> http://flink.apache.org/contribute-code.html.
>> > >> >> >> > > > >>> > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > -
>> > >> >> >> > > > >>> > > > > > > > Aljoscha
>> > >> >> >> > > > >>> > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > On Wed, 6 Jul 2016 at 19:45 Maxim <
>> > >> >> >> mfat...@gmail.com
>> > >> >> >> > >
>> > >> >> >> > > > >>> wrote:
>> > >> >> >> > > > >>> > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > The new API forces iteration through
>> > every
>> > >> >> >> element
>> > >> >> >> > of
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> > > buffer
>> > >> >> >> > > > >>> > > > > > > > > even
>> > >> >> >> > > > >>> > > > > > > if
>> > >> >> >> > > > >>> > > > > > > > a
>> > >> >> >> > > > >>> > > > > > > > > single value to be evicted. What
>> about
>> > >> >> >> implementing
>> > >> >> >> > > > >>> > > > > > > > > Iterator.remove() method for
>> elements?
>> > The
>> > >> >> API
>> > >> >> >> > would
>> > >> >> >> > > > look
>> > >> >> >> > > > >>> > like:
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > public interface Evictor<T, W extends
>> > >> Window>
>> > >> >> >> > extends
>> > >> >> >> > > > >>> > > > > > > > > Serializable {
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > /**
>> > >> >> >> > > > >>> > > > > > > > > * Optionally evicts elements.
>> > Called
>> > >> >> before
>> > >> >> >> > > > >>> windowing
>> > >> >> >> > > > >>> > > > > function.
>> > >> >> >> > > > >>> > > > > > > > > *
>> > >> >> >> > > > >>> > > > > > > > > * @param elements The elements
>> > >> currently
>> > >> >> in
>> > >> >> >> the
>> > >> >> >> > > > >>> pane. Use
>> > >> >> >> > > > >>> > > > > > > > > Iterator.remove to evict.
>> > >> >> >> > > > >>> > > > > > > > > * @param size The current number
>> of
>> > >> >> >> elements in
>> > >> >> >> > > the
>> > >> >> >> > > > >>> pane.
>> > >> >> >> > > > >>> > > > > > > > > * @param window The {@link
>> Window}
>> > >> >> >> > > > >>> > > > > > > > > */
>> > >> >> >> > > > >>> > > > > > > > > void evictBefore(Iterable<T>
>> > elements,
>> > >> int
>> > >> >> >> size,
>> > >> >> >> > > > >>> > > > > > > > > EvictorContext
>> > >> >> >> > > > >>> > > > > > > ctx);
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > /**
>> > >> >> >> > > > >>> > > > > > > > > * Optionally evicts elements.
>> > Called
>> > >> >> after
>> > >> >> >> > > > windowing
>> > >> >> >> > > > >>> > > > function.
>> > >> >> >> > > > >>> > > > > > > > > *
>> > >> >> >> > > > >>> > > > > > > > > * @param elements The elements
>> > >> currently
>> > >> >> in
>> > >> >> >> the
>> > >> >> >> > > > >>> pane. Use
>> > >> >> >> > > > >>> > > > > > > > > Iterator.remove to evict.
>> > >> >> >> > > > >>> > > > > > > > > * @param size The current number
>> of
>> > >> >> >> elements in
>> > >> >> >> > > the
>> > >> >> >> > > > >>> pane.
>> > >> >> >> > > > >>> > > > > > > > > * @param window The {@link
>> Window}
>> > >> >> >> > > > >>> > > > > > > > > */
>> > >> >> >> > > > >>> > > > > > > > > void evictAfter(Iterable<T>
>> elements,
>> > >> int
>> > >> >> >> size,
>> > >> >> >> > > > >>> > > > > > > > > EvictorContext ctx); }
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > Such API allows to abort iteration at
>> > any
>> > >> >> point
>> > >> >> >> and
>> > >> >> >> > > > evict
>> > >> >> >> > > > >>> > > > > > > > > elements in
>> > >> >> >> > > > >>> > > > > > > any
>> > >> >> >> > > > >>> > > > > > > > > order.
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > Thanks,
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > Maxim.
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > On Wed, Jul 6, 2016 at 9:04 AM,
>> Vishnu
>> > >> >> >> Viswanath <
>> > >> >> >> > > > >>> > > > > > > > > vishnu.viswanat...@gmail.com> wrote:
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > Hi Aljoscha,
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > Thanks. Yes the new interface
>> seems to
>> > >> >> address
>> > >> >> >> > > points
>> > >> >> >> > > > >>> 1 and
>> > >> >> >> > > > >>> > > 2.
>> > >> >> >> > > > >>> > > > > > > > > > of
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > *1) I am having a use case where I
>> > have
>> > >> to
>> > >> >> >> > create a
>> > >> >> >> > > > >>> custom
>> > >> >> >> > > > >>> > > > > > > > > > Evictor
>> > >> >> >> > > > >>> > > > > > > that
>> > >> >> >> > > > >>> > > > > > > > > > will evict elements from the window
>> > >> based
>> > >> >> on
>> > >> >> >> the
>> > >> >> >> > > > value
>> > >> >> >> > > > >>> > (e.g.,
>> > >> >> >> > > > >>> > > > > > > > > > if I
>> > >> >> >> > > > >>> > > > > > > have
>> > >> >> >> > > > >>> > > > > > > > > > elements are of case class Item(id:
>> > Int,
>> > >> >> >> > > type:String)
>> > >> >> >> > > > >>> then
>> > >> >> >> > > > >>> > > > > > > > > > evict
>> > >> >> >> > > > >>> > > > > > > > elements
>> > >> >> >> > > > >>> > > > > > > > > > that has type="a"). I believe this
>> is
>> > >> not
>> > >> >> >> > currently
>> > >> >> >> > > > >>> > > possible.*
>> > >> >> >> > > > >>> > > > > > > > > > *2) this is somewhat related to 1)
>> > where
>> > >> >> there
>> > >> >> >> > > should
>> > >> >> >> > > > >>> be an
>> > >> >> >> > > > >>> > > > > > > > > > option to
>> > >> >> >> > > > >>> > > > > > > > > evict
>> > >> >> >> > > > >>> > > > > > > > > > elements from anywhere in the
>> window.
>> > >> not
>> > >> >> only
>> > >> >> >> > from
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> > > > > > > > > > beginning of
>> > >> >> >> > > > >>> > > > > > > > the
>> > >> >> >> > > > >>> > > > > > > > > > window. (e.g., apply the delta
>> > function
>> > >> to
>> > >> >> all
>> > >> >> >> > > > >>> elements and
>> > >> >> >> > > > >>> > > > > > > > > > remove
>> > >> >> >> > > > >>> > > > > > > all
>> > >> >> >> > > > >>> > > > > > > > > > those don't pass. I checked the
>> code
>> > and
>> > >> >> evict
>> > >> >> >> > > method
>> > >> >> >> > > > >>> just
>> > >> >> >> > > > >>> > > > > > > > > > returns
>> > >> >> >> > > > >>> > > > > > > the
>> > >> >> >> > > > >>> > > > > > > > > > number of elements to be removed
>> and
>> > >> >> >> > > > >>> processTriggerResult
>> > >> >> >> > > > >>> > > just
>> > >> >> >> > > > >>> > > > > > > > > > skips
>> > >> >> >> > > > >>> > > > > > > > > those
>> > >> >> >> > > > >>> > > > > > > > > > many elements from the beginning.
>> *
>> > >> >> >> > > > >>> > > > > > > > > > *3) Add an option to enables the
>> user
>> > to
>> > >> >> >> decide
>> > >> >> >> > if
>> > >> >> >> > > > the
>> > >> >> >> > > > >>> > > > > > > > > > eviction
>> > >> >> >> > > > >>> > > > > > > should
>> > >> >> >> > > > >>> > > > > > > > > > happen before the apply function or
>> > >> after
>> > >> >> the
>> > >> >> >> > apply
>> > >> >> >> > > > >>> > function.
>> > >> >> >> > > > >>> > > > > > > Currently
>> > >> >> >> > > > >>> > > > > > > > > it
>> > >> >> >> > > > >>> > > > > > > > > > is before the apply function, but I
>> > >> have a
>> > >> >> use
>> > >> >> >> > case
>> > >> >> >> > > > >>> where I
>> > >> >> >> > > > >>> > > > > > > > > > need to
>> > >> >> >> > > > >>> > > > > > > > first
>> > >> >> >> > > > >>> > > > > > > > > > apply the function and evict
>> > afterward.*
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > I would be interested in
>> contributing
>> > to
>> > >> >> the
>> > >> >> >> code
>> > >> >> >> > > > base.
>> > >> >> >> > > > >>> > > Please
>> > >> >> >> > > > >>> > > > > > > > > > let me
>> > >> >> >> > > > >>> > > > > > > > > know
>> > >> >> >> > > > >>> > > > > > > > > > the steps.
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > Thanks and Regards,
>> > >> >> >> > > > >>> > > > > > > > > > Vishnu Viswanath
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > On Wed, Jul 6, 2016 at 11:49 AM,
>> > >> Aljoscha
>> > >> >> >> > Krettek <
>> > >> >> >> > > > >>> > > > > > > aljos...@apache.org
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > wrote:
>> > >> >> >> > > > >>> > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > > Hi,
>> > >> >> >> > > > >>> > > > > > > > > > > as mentioned in the thread on
>> > >> improving
>> > >> >> the
>> > >> >> >> > > > Windowing
>> > >> >> >> > > > >>> > API I
>> > >> >> >> > > > >>> > > > > > > > > > > also
>> > >> >> >> > > > >>> > > > > > > > have a
>> > >> >> >> > > > >>> > > > > > > > > > > design doc just for improving
>> > >> >> >> WindowEvictors. I
>> > >> >> >> > > had
>> > >> >> >> > > > >>> this
>> > >> >> >> > > > >>> > in
>> > >> >> >> > > > >>> > > > > > > > > > > my head
>> > >> >> >> > > > >>> > > > > > > > for
>> > >> >> >> > > > >>> > > > > > > > > a
>> > >> >> >> > > > >>> > > > > > > > > > > while but was hesitant to publish
>> > but
>> > >> >> since
>> > >> >> >> > > people
>> > >> >> >> > > > >>> are
>> > >> >> >> > > > >>> > > > > > > > > > > asking about
>> > >> >> >> > > > >>> > > > > > > > > this
>> > >> >> >> > > > >>> > > > > > > > > > > now might be a good time to post
>> it.
>> > >> >> Here's
>> > >> >> >> the
>> > >> >> >> > > > doc:
>> > >> >> >> > > > >>> > > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > >
>> > >> >> >> > > > >>> > > > > > >
>> > >> >> >> > > > >>> > >
>> > >> >> >> > > >
>> > >> >> >>
>> > >> https://docs.google.com/document/d/1Rr7xzlItYqvFXLyyy-Yv0vvw8f29QYAj
>> > >> >> >> > > > >>> > > > > > > m5
>> > >> >> >> > > > >>> > > > > > > i9E4A_JlU/edit?usp=sharing
>> > >> >> >> > > > >>> > > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > > Feedback/Suggestions are very
>> > welcome!
>> > >> >> >> Please
>> > >> >> >> > let
>> > >> >> >> > > > me
>> > >> >> >> > > > >>> know
>> > >> >> >> > > > >>> > > > > > > > > > > what you
>> > >> >> >> > > > >>> > > > > > > > > think.
>> > >> >> >> > > > >>> > > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > > @Vishnu: Are you interested in
>> > >> >> contributing
>> > >> >> >> a
>> > >> >> >> > > > >>> solution
>> > >> >> >> > > > >>> > for
>> > >> >> >> > > > >>> > > > > > > > > > > this to
>> > >> >> >> > > > >>> > > > > > > > the
>> > >> >> >> > > > >>> > > > > > > > > > > Flink code base? I'd be very
>> happy
>> > to
>> > >> >> work
>> > >> >> >> with
>> > >> >> >> > > you
>> > >> >> >> > > > >>> on
>> > >> >> >> > > > >>> > > this.
>> > >> >> >> > > > >>> > > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > > Cheers,
>> > >> >> >> > > > >>> > > > > > > > > > > Aljoscha
>> > >> >> >> > > > >>> > > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > > > > P.S. I think it would be best to
>> > keep
>> > >> >> >> > discussions
>> > >> >> >> > > > to
>> > >> >> >> > > > >>> the
>> > >> >> >> > > > >>> > ML
>> > >> >> >> > > > >>> > > > > > > > > > > because comments on the doc will
>> not
>> > >> be
>> > >> >> >> visible
>> > >> >> >> > > > here
>> > >> >> >> > > > >>> for
>> > >> >> >> > > > >>> > > > > > everyone.
>> > >> >> >> > > > >>> > > > > > > > > > >
>> > >> >> >> > > > >>> > > > > > > > >
>> > >> >> >> > > > >>> > > > > > >
>>
>
>
>
