Actually for such evictor to be useful the window should be sorted by some field, usually event time. What do you think about adding sorted window abstraction?
On Wed, Jul 6, 2016 at 11:36 AM, Aljoscha Krettek <aljos...@apache.org> wrote: > @Maxim: That's perfect I didn't think about using Iterator.remove() for > that. I'll update the doc. What do you think Vishnu? This should also cover > your before/after case nicely. > > @Vishnu: The steps would be these: > - Converge on a design in this discussion > - Add a Jira issue here: https://issues.apache.org/jira/browse/FLINK > - Work on the code an create a pull request on github > > The steps are also outlined here > http://flink.apache.org/how-to-contribute.html and here > http://flink.apache.org/contribute-code.html. > > - > Aljoscha > > On Wed, 6 Jul 2016 at 19:45 Maxim <mfat...@gmail.com> wrote: > > > The new API forces iteration through every element of the buffer even if > a > > single value to be evicted. What about implementing Iterator.remove() > > method for elements? The API would look like: > > > > public interface Evictor<T, W extends Window> extends Serializable { > > > > /** > > * Optionally evicts elements. Called before windowing function. > > * > > * @param elements The elements currently in the pane. Use > > Iterator.remove to evict. > > * @param size The current number of elements in the pane. > > * @param window The {@link Window} > > */ > > void evictBefore(Iterable<T> elements, int size, EvictorContext ctx); > > > > /** > > * Optionally evicts elements. Called after windowing function. > > * > > * @param elements The elements currently in the pane. Use > > Iterator.remove to evict. > > * @param size The current number of elements in the pane. > > * @param window The {@link Window} > > */ > > void evictAfter(Iterable<T> elements, int size, EvictorContext ctx); > > } > > > > Such API allows to abort iteration at any point and evict elements in any > > order. > > > > Thanks, > > > > Maxim. > > > > On Wed, Jul 6, 2016 at 9:04 AM, Vishnu Viswanath < > > vishnu.viswanat...@gmail.com> wrote: > > > > > > Hi Aljoscha, > > > > > > Thanks. Yes the new interface seems to address points 1 and 2. of > > > > > > *1) I am having a use case where I have to create a custom Evictor that > > > will evict elements from the window based on the value (e.g., if I have > > > elements are of case class Item(id: Int, type:String) then evict > elements > > > that has type="a"). I believe this is not currently possible.* > > > *2) this is somewhat related to 1) where there should be an option to > > evict > > > elements from anywhere in the window. not only from the beginning of > the > > > window. (e.g., apply the delta function to all elements and remove all > > > those don't pass. I checked the code and evict method just returns the > > > number of elements to be removed and processTriggerResult just skips > > those > > > many elements from the beginning. * > > > *3) Add an option to enables the user to decide if the eviction should > > > happen before the apply function or after the apply function. Currently > > it > > > is before the apply function, but I have a use case where I need to > first > > > apply the function and evict afterward.* > > > > > > I would be interested in contributing to the code base. Please let me > > know > > > the steps. > > > > > > Thanks and Regards, > > > Vishnu Viswanath > > > > > > On Wed, Jul 6, 2016 at 11:49 AM, Aljoscha Krettek <aljos...@apache.org > > > > > wrote: > > > > > > > Hi, > > > > as mentioned in the thread on improving the Windowing API I also > have a > > > > design doc just for improving WindowEvictors. I had this in my head > for > > a > > > > while but was hesitant to publish but since people are asking about > > this > > > > now might be a good time to post it. Here's the doc: > > > > > > > > > https://docs.google.com/document/d/1Rr7xzlItYqvFXLyyy-Yv0vvw8f29QYAjm5i9E4A_JlU/edit?usp=sharing > > > > > > > > Feedback/Suggestions are very welcome! Please let me know what you > > think. > > > > > > > > @Vishnu: Are you interested in contributing a solution for this to > the > > > > Flink code base? I'd be very happy to work with you on this. > > > > > > > > Cheers, > > > > Aljoscha > > > > > > > > P.S. I think it would be best to keep discussions to the ML because > > > > comments on the doc will not be visible here for everyone. > > > > > > >
