Could you point me towards the inspiration for Evictors? Are there any papers, perhaps, that lay the groundwork for mutable windows like this?
After much research this weekend, I found that Evictors are unique to Flink. Conceptually, it looks to me like Dataflow windows are build-only. Looking into other Dataflow implementations: I didn't find anything in either the Apache Beam SDK docs or the Google Cloud Dataflow API docs that mention allowing you to remove elements from a window. I'm hesitant to tread new ground in mutability. What do you think about reimplementing Evictors as a kind of cyclic filter operation? Would it be possible? I believe this would fit into the Dataflow model better, but I'm still in the early stages of becoming familiar with Flink, and I haven't read the ABS paper [1] yet to know if there are snapshot implications. I also don't (yet) see why you couldn't optimize such a cyclic operation with mutable operations under the hood. [1]: http://arxiv.org/abs/1506.08603 On Fri, Sep 9, 2016 at 11:46 AM, AJ Heller <a...@drfloob.com> wrote: > Thank you for offering your support, I'm excited to dig in! > > I have some work to do getting up to speed on the windowing internals. And > I still need to get my bearing on the Evictor changes, I plan to read > through the list archive and documents today. Vishnu, are your changes > already publicly viewable? > > Regarding the window modifications in FLIP-2, I see Vishnu that you've > suggested an interface for the EvictorContext object, and Aljoscha, you > suggested an abstract Context class. Does it make sense for them to agree? > The other big difference I've seen in the signatures is wheter the Window > is contained in the context or not. > > Have you considered modifying the signature of the methods to accept `<C > extends Context>` or `<EC extends EvictorContext>`? At least in terms of > FLIP-2, this would allow each process window function to define and work > with its own context (without downcasting, anyway), and similarly in the > future, there'd be less work in changing Context subclasses when new > abstract methods are added to Context. > > But I may be getting ahead of myself. Could you point me towards where > contexts are/would be created? I'm not clear on the ownership and lifecycle > of these objects yet. >
