Re: Implementing FLIP-2 and FLIP-4

2016-09-17 Thread Aljoscha Krettek
Hi AJ, sorry for not getting back to you earlier, I was too busy and only read your mail now. Adding the context is not a simple optimization of the case you described. In your case, you will get 30-second windows where the elements are assigned to those windows based on their timestamp. If you ha

Re: Implementing FLIP-2 and FLIP-4

2016-09-14 Thread AJ Heller
Thank you, Aljoscha! I look forward to reading the papers you mentioned. Regarding FLIP-2, are there any new use cases that a Window Function Context enables? If not, my understanding is that adding a this context would be an optimization over what is currently possible, but maybe inefficient. For

Re: Implementing FLIP-2 and FLIP-4

2016-09-14 Thread Aljoscha Krettek
Hi AJ, the idea for evictors initially came from IBM Infosphere Streams, if I'm not mistaken: http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.dev.doc/doc/windowhandling.html The first version of the windowing system used a combination of triggers/evictors to do the windowing

Re: Implementing FLIP-2 and FLIP-4

2016-09-11 Thread AJ Heller
Could you point me towards the inspiration for Evictors? Are there any papers, perhaps, that lay the groundwork for mutable windows like this? After much research this weekend, I found that Evictors are unique to Flink. Conceptually, it looks to me like Dataflow windows are build-only. Looking int

Implementing FLIP-2 and FLIP-4

2016-09-09 Thread AJ Heller
Thank you for offering your support, I'm excited to dig in! I have some work to do getting up to speed on the windowing internals. And I still need to get my bearing on the Evictor changes, I plan to read through the list archive and documents today. Vishnu, are your changes already publicly viewa