Hi Rong, Thanks for the improvement proposal, this topic aroused my interest since we did some similar improvements in Blink. After going through your design doc, i would like share some thoughts:
1. It looks to me the proposed SliceManager and MergeManager is quite generic and can be used in various situations like unaligned windows. However this will introduce some new API and some new stream operators, and thus make is a big effort to do. I suppose the main reason to introduce these concepts is we want to support some complex scenarios like unaligned windows, multi timeout session windows and so on. But if let's go back to the original motivation of this improvement, it will be "Highly overlapping window operation" and "Window functions with efficient methods calculating and merging partial results", just as you mentioned in the section "Target Usage Pattern". I'm curious if it's necessary to introduce all these new APIs and operators to meet these requirements. If i understand you correctly, we may even can achieve the goal by using two successive windows, first with tumbling window, and then sliding window. And from the experiences we had in Blink, by adding some enhancements to original window operator may also be sufficient. Does it make sense to you if we can first try to improve the target scenario without adding new APIs and operators? It will definitely make it more easy to review and merged. 2. From what we observed after Blink did similar improvements, slice or pane based improvement is not a silver bullet for all kinds of situations. In most cases, we only observed small performance gain like 20% or 30%. The main reason is, if we want get big gain from this improvement, normally we require the window function have efficient calculating and merging methods, like SUM, COUNT. If this assumption stands, the basic approach of window operator will also be performant. It stored compact intermedia result for each window per key, and it's also very efficient to get result based on this. The only downside is we need to do the calculation in multiply windows per key, but based on the assumption, the calculation is efficient and may not be a blocker. Based on this, i think we must be more careful about the changes, especially API related. 3. The last thing in my mind is whether we can share some implementations with blink's window operator. We introduced a new window operator in Blink SQL, which i think needs to be discussed when we try to adopt blink improvements. Without strong reasons, i think it's better to share some implementations with DataStream's window operator. Maybe we don't need to share the whole operator, but at least some underlying concepts and data structures. It would also be great if you can think about it and see if it's a feasible way. Last small question: how do you handle sliding windows whose window size can't be divided by step, such as 10 seconds window and slide with 3 seconds? Best, Kurt On Wed, Jan 30, 2019 at 2:12 AM Fabian Hueske <fhue...@gmail.com> wrote: > Thank you Rong! > The performance of sliding windows is an issue for many users. > Adding support for a more efficient window is a great effort. > > Thank you, > Fabian > > Am Di., 29. Jan. 2019 um 16:37 Uhr schrieb Rong Rong <walter...@gmail.com > >: > > > Hi all, > > > > Thanks for the feedbacks and suggestions to the design doc. I have > created > > a parent JIRA [1] to track the related tasks and started the > implementation > > process > > Any further feedbacks or suggestions are highly appreciated. > > > > Best, > > Rong > > > > [1] https://issues.apache.org/jira/browse/FLINK-11276 > > > > On Wed, Dec 5, 2018 at 6:17 PM Rong Rong <walter...@gmail.com> wrote: > > > > > Hi all, > > > > > > Various discussion in the mailing list & JIRA tickets [2] had been > > brought > > > up in the past regarding the windowing operation performance. As we > > > experiment internally with some of our extreme use cases, we found out > > that > > > using a slice-based implementation can optimize Flink's windowing > > mechanism > > > and provide a better performance in most cases. > > > > > > We've put together a preliminary enhancement and performance > optimization > > > plan [1] for the current windowing operation in Flink. This is largely > > > inspired by stream slicing research shared in recent Flink Forward > > > conference [3] by Philip and Jonas, and the discussion in the main JIRA > > > ticket [2]. The initial design and POC implementations consider > > optimizing > > > the performance for the category of overlapping windows as well as > > allowing > > > chaining of cascade window operators. > > > > > > It will be great to hear the feedbacks and suggestions from the > > community. > > > Please kindly share your comments and suggestions. > > > > > > Thanks, > > > Rong > > > > > > [1] > > > > > > https://docs.google.com/document/d/1ziVsuW_HQnvJr_4a9yKwx_LEnhVkdlde2Z5l6sx5HlY/edit?usp=sharing > > > [2] https://issues.apache.org/jira/browse/FLINK-7001 > > > [3] > > > > > > https://data-artisans.com/flink-forward-berlin/resources/efficient-window-aggregation-with-stream-slicing > > > > > > > > > > > > > > >
