Hi Gary, Thanks for kicking off the features for next release 1.10. I am very supportive of you and Yu Li to be the relaese managers.
Just mention another two improvements which want to be covered in FLINK-1.10 and I already confirmed with Piotr to reach an agreement before. 1. Data serialize and copy only once for broadcast partition [1]: It would improve the throughput performance greatly in broadcast mode and was actually proposed in Flink-1.8. Most of works already done before and only left the last critical jira/PR. It will not take much efforts to make it ready. 2. Let Netty use Flink's buffers directly in credit-based mode [2] : It could avoid memory copy from netty stack to flink managed network buffer. The obvious benefit is decreasing the direct memory overhead greatly in large-scale jobs. I also heard of some user cases encounter direct OOM caused by netty memory overhead. Actually this improvment was proposed by nico in FLINK-1.7 and always no time to focus then. Yun Gao already submitted a PR half an year ago but have not been reviewed yet. I could help review the deign and PR codes to make it ready. And you could make these two items as lowest priority if possible. [1] https://issues.apache.org/jira/browse/FLINK-10745 [2] https://issues.apache.org/jira/browse/FLINK-10742 Best, Zhijiang ------------------------------------------------------------------ From:Gary Yao <g...@apache.org> Send Time:2019年9月6日(星期五) 17:06 To:dev <dev@flink.apache.org> Cc:carp84 <car...@gmail.com> Subject:[DISCUSS] Features for Apache Flink 1.10 Hi community, Since Apache Flink 1.9.0 has been released more than 2 weeks ago, I want to start kicking off the discussion about what we want to achieve for the 1.10 release. Based on discussions with various people as well as observations from mailing list threads, Yu Li and I have compiled a list of features that we deem important to be included in the next release. Note that the features presented here are not meant to be exhaustive. As always, I am sure that there will be other contributions that will make it into the next release. This email thread is merely to kick off a discussion, and to give users and contributors an understanding where the focus of the next release lies. If there is anything we have missed that somebody is working on, please reply to this thread. ** Proposed features and focus Following the contribution of Blink to Apache Flink, the community released a preview of the Blink SQL Query Processor, which offers better SQL coverage and improved performance for batch queries, in Flink 1.9.0. However, the integration of the Blink query processor is not fully completed yet as there are still pending tasks, such as implementing full TPC-DS support. With the next Flink release, we aim at finishing the Blink integration. Furthermore, there are several ongoing work threads addressing long-standing issues reported by users, such as improving checkpointing under backpressure, and limiting RocksDBs native memory usage, which can be especially problematic in containerized Flink deployments. Notable features surrounding Flink’s ecosystem that are planned for the next release include active Kubernetes support (i.e., enabling Flink’s ResourceManager to launch new pods), improved Hive integration, Java 11 support, and new algorithms for the Flink ML library. Below I have included the list of features that we compiled ordered by priority – some of which already have ongoing mailing list threads, JIRAs, or FLIPs. - Improving Flink’s build system & CI [1] [2] - Support Java 11 [3] - Table API improvements - Configuration Evolution [4] [5] - Finish type system: Expression Re-design [6] and UDF refactor - Streaming DDL: Time attribute (watermark) and Changelog support - Full SQL partition support for both batch & streaming [7] - New Java Expression DSL [8] - SQL CLI with DDL and DML support - Hive compatibility completion (DDL/UDF) to support full Hive integration - Partition/Function/View support - Remaining Blink planner/runtime merge - Support all TPC-DS queries [9] - Finer grained resource management - Unified TaskExecutor Memory Configuration [10] - Fine Grained Operator Resource Management [11] - Dynamic Slots Allocation [12] - Finish scheduler re-architecture [13] - Allows implementing more sophisticated scheduling strategies such as better batch scheduler or speculative execution. - New DataStream Source Interface [14] - A new source connector architecture to unify the implementation of source connectors and make it simpler to implement custom source connectors. - Add more source/system metrics - For better flink job monitoring and facilitate customized solutions like auto-scaling. - Executor Interface / Client API [15] - Allow Flink downstream projects to easier and better monitor and control flink jobs. - Interactive Programming [16] - Allow users to cache the intermediate results in Table API for later usage to avoid redundant computation when a Flink application contains multiple jobs. - Python User Defined Function [17] - Support native user-defined functions in Flink Python, including UDF/UDAF/UDTF in Table API and Python-Java mixed UDF. - Spillable heap backend [18] - A new state backend supporting automatic data spill and load when memory exhausted/regained. - RocksDB backend memory control [19] - Prevent excessive memory usage from RocksDB, especially in container environment. - Unaligned checkpoints [20] - Resolve the checkpoint timeout issue under backpressure. - Separate framework and user class loader in per-job mode - Active Kubernetes Integration [21] - Allow ResourceManager talking to Kubernetes to launch new pods similar to Flink's Yarn/Mesos integration - ML pipeline/library - Aims at delivering several core algorithms, including Logistic Regression, Native Bayes, Random Forest, KMeans, etc. - Add vertex subtask log url on WebUI [22] ** Suggested release timeline Based on our usual time-based release schedule [23], and considering that several events, such as Flink Forward Europe and Asia, are overlapping with the current release cycle, we should aim at releasing 1.10 around the beginning of January 2020. To give the community enough testing time, I propose the feature freeze to be at the end of November. We should announce an exact date later in the release cycle. Lastly, I would like to use the opportunity to propose Yu Li and myself as release managers for the upcoming release. What do you think? Best, Gary [1] https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E [2] https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E [3] https://issues.apache.org/jira/browse/FLINK-10725 [4] https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration [5] https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object [6] https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design [7] https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support [8] https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL [9] https://issues.apache.org/jira/browse/FLINK-11491 [10] https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors [11] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management [12] https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation [13] https://issues.apache.org/jira/browse/FLINK-10429 [14] https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface [15] https://lists.apache.org/thread.html/498dd3e0277681cda356029582c1490299ae01df912e15942e11ae8e@%3Cdev.flink.apache.org%3E [16] https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink [17] https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table [18] https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend [19] https://issues.apache.org/jira/browse/FLINK-7289 [20] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html [21] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html [22] https://issues.apache.org/jira/browse/FLINK-13894 [23] https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
