Hi Stephan, Thanks for bringing up the discussions. I'm +1 on the merging plan. One question though: since the merge will not be completed for some time and there are might be uses trying blink branch, what's the plan for the development in the branch? Personally I think we may discourage big contributions to the branch, which would further complicate the merge, while we shouldn't stop critical fixes as well.
What's your take on this? Thanks, Xuefu ------------------------------------------------------------------ From:Stephan Ewen <se...@apache.org> Sent At:2019 Jan. 22 (Tue.) 06:16 To:dev <dev@flink.apache.org> Subject:[DISCUSS] A strategy for merging the Blink enhancements Dear Flink community! As a follow-up to the thread announcing Alibaba's offer to contribute the Blink code [1] <https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E> , here are some thoughts on how this contribution could be merged. As described in the announcement thread, it is a big contribution, and we need to carefully plan how to handle the contribution. We would like to get the improvements to Flink, while making it as non-disruptive as possible for the community. I hope that this plan gives the community get a better understanding of what the proposed contribution would mean. Here is an initial rough proposal, with thoughts from Timo, Piotr, Dawid, Kurt, Shaoxuan, Jincheng, Jark, Aljoscha, Fabian, Xiaowei: - It is obviously very hard to merge all changes in a quick move, because we are talking about multiple 100k lines of code. - As much as possible, we want to maintain compatibility with the current Table API, so that this becomes a transparent change for most users. - The two areas with the most changes we identified were (1) The SQL/Table query processor (2) The batch scheduling/failover/shuffle - For the query processor part, this is what we found and propose: -> The Blink and Flink code have the same semantics (ANSI SQL) except for minor aspects (under discussion). Blink also covers more SQL operations. -> The Blink code is quite different from the current Flink SQL runtime. Merging as changes seems hardly feasible. From the current evaluation, the Blink query processor uses the more advanced architecture, so it would make sense to converge to that design. -> We propose to gradually build up the Blink-based query processor as a second query processor under the SQL/Table API. Think of it as two different runners for the Table API. As the new query processor becomes fully merged and stable, we can deprecate and eventually remove the existing query processor. That should give the least disruption to Flink users and allow for gradual merge/development. -> Some refactoring of the Table API is necessary to support the above strategy. Most of the prerequisite refactoring is around splitting the project into different modules, following a similar idea as FLIP-28 [2] <https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free> . -> A more detailed proposal is being worked on. -> Same as FLIP-28, this approach would probably need to suspend Table API contributions for a short while. We hope that this can be a very short period, to not impact the very active development in Flink on Table API/SQL too much. - For the batch scheduling and failover enhancements, we should be able to build on the currently ongoing refactoring of the scheduling logic [3] <https://issues.apache.org/jira/browse/FLINK-10429>. That should make it easy to plug in a new scheduler and failover logic. We can port the Blink enhancements as a new scheduler / failover handler. We can later make it the default for bounded stream programs once the merge is completed and it is tested. - For the catalog and source/sink design and interfaces, we would like to continue with the already started design discussion threads. Once these are converged, we might use some of the Blink code for the implementation, if it is close to the outcome of the design discussions. Best, Stephan [1] https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free [3] https://issues.apache.org/jira/browse/FLINK-10429
