Re: Consistent (hashing) keyBy over multiple time or streaming windows

2015-11-02 Thread Leonard Wolters
Hi Aljoscha, Thanks for the quick response. I've seen the Google Data Flow presentation @ Flink forward and understand the concepts behind it (which are also supported by Flink). I will further look into stack overflow and let you know if I have some further questions. Once again, thanks,

Re: Consistent (hashing) keyBy over multiple time or streaming windows

2015-11-02 Thread Aljoscha Krettek
Hi Leonard, I’m afraid you might be thinking about windows as they are supported by Spark Streaming. There windows are quite limited. In Flink you don’t necessarily have to window elements by time since Flink does not collect data in mini-batches before processing. Everything is continuously pro

Consistent (hashing) keyBy over multiple time or streaming windows

2015-11-02 Thread Leonard Wolters
Hi, I was wondering if Flink already has implemented some sort of consistent keyBy mapping over multiple windows. The underlying idea is to 'sessionize' incoming events over time (i.e. multiple streaming windows) on the same partitions. As one can understand I want to avoid heavy shuffling over