Thanks for compiling the list of 1.10 efforts for the community Gary. I think this helps a lot to better understand what the community is currently working on.
Thanks for volunteering as the release managers for the next major release. +1 for Gary and Yu being the RMs for Flink 1.10. Cheers, Till On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu <reed...@gmail.com> wrote: > Thanks Gary for kicking off this discussion. > Really appreciate that you and Yu offer to help to manage 1.10 release. > > +1 for Gary and Yu as release managers. > > Thanks, > Zhu Zhu > > Dian Fu <dian0511...@gmail.com> 于2019年9月7日周六 下午12:26写道： > > > Hi Gary, > > > > Thanks for kicking off the release schedule of 1.10. +1 for you and Yu Li > > as the release manager. > > > > The feature freeze/release time sounds reasonable. > > > > Thanks, > > Dian > > > > > 在 2019年9月7日，上午11:30，Jark Wu <imj...@gmail.com> 写道： > > > > > > Thanks Gary for kicking off the discussion for 1.10 release. > > > > > > +1 for Gary and Yu as release managers. Thank you for you effort. > > > > > > Best, > > > Jark > > > > > > > > >> 在 2019年9月7日，00:52，zhijiang <wangzhijiang...@aliyun.com.INVALID> 写道： > > >> > > >> Hi Gary, > > >> > > >> Thanks for kicking off the features for next release 1.10. I am very > > supportive of you and Yu Li to be the relaese managers. > > >> > > >> Just mention another two improvements which want to be covered in > > FLINK-1.10 and I already confirmed with Piotr to reach an agreement > before. > > >> > > >> 1. Data serialize and copy only once for broadcast partition [1]: It > > would improve the throughput performance greatly in broadcast mode and > was > > actually proposed in Flink-1.8. Most of works already done before and > only > > left the last critical jira/PR. It will not take much efforts to make it > > ready. > > >> > > >> 2. Let Netty use Flink's buffers directly in credit-based mode [2] : > It > > could avoid memory copy from netty stack to flink managed network buffer. > > The obvious benefit is decreasing the direct memory overhead greatly in > > large-scale jobs. I also heard of some user cases encounter direct OOM > > caused by netty memory overhead. Actually this improvment was proposed by > > nico in FLINK-1.7 and always no time to focus then. Yun Gao already > > submitted a PR half an year ago but have not been reviewed yet. I could > > help review the deign and PR codes to make it ready. > > >> > > >> And you could make these two items as lowest priority if possible. > > >> > > >> [1] https://issues.apache.org/jira/browse/FLINK-10745 > > >> [2] https://issues.apache.org/jira/browse/FLINK-10742 > > >> > > >> Best, > > >> Zhijiang > > >> ------------------------------------------------------------------ > > >> From:Gary Yao <g...@apache.org> > > >> Send Time:2019年9月6日(星期五) 17:06 > > >> To:dev <dev@flink.apache.org> > > >> Cc:carp84 <car...@gmail.com> > > >> Subject:[DISCUSS] Features for Apache Flink 1.10 > > >> > > >> Hi community, > > >> > > >> Since Apache Flink 1.9.0 has been released more than 2 weeks ago, I > > want to > > >> start kicking off the discussion about what we want to achieve for the > > 1.10 > > >> release. > > >> > > >> Based on discussions with various people as well as observations from > > >> mailing > > >> list threads, Yu Li and I have compiled a list of features that we > deem > > >> important to be included in the next release. Note that the features > > >> presented > > >> here are not meant to be exhaustive. As always, I am sure that there > > will be > > >> other contributions that will make it into the next release. This > email > > >> thread > > >> is merely to kick off a discussion, and to give users and contributors > > an > > >> understanding where the focus of the next release lies. If there is > > anything > > >> we have missed that somebody is working on, please reply to this > thread. > > >> > > >> > > >> ** Proposed features and focus > > >> > > >> Following the contribution of Blink to Apache Flink, the community > > released > > >> a > > >> preview of the Blink SQL Query Processor, which offers better SQL > > coverage > > >> and > > >> improved performance for batch queries, in Flink 1.9.0. However, the > > >> integration of the Blink query processor is not fully completed yet as > > there > > >> are still pending tasks, such as implementing full TPC-DS support. > With > > the > > >> next Flink release, we aim at finishing the Blink integration. > > >> > > >> Furthermore, there are several ongoing work threads addressing > > long-standing > > >> issues reported by users, such as improving checkpointing under > > >> backpressure, > > >> and limiting RocksDBs native memory usage, which can be especially > > >> problematic > > >> in containerized Flink deployments. > > >> > > >> Notable features surrounding Flink’s ecosystem that are planned for > the > > next > > >> release include active Kubernetes support (i.e., enabling Flink’s > > >> ResourceManager to launch new pods), improved Hive integration, Java > 11 > > >> support, and new algorithms for the Flink ML library. > > >> > > >> Below I have included the list of features that we compiled ordered by > > >> priority – some of which already have ongoing mailing list threads, > > JIRAs, > > >> or > > >> FLIPs. > > >> > > >> - Improving Flink’s build system & CI [1] [2] > > >> - Support Java 11 [3] > > >> - Table API improvements > > >> - Configuration Evolution [4] [5] > > >> - Finish type system: Expression Re-design [6] and UDF refactor > > >> - Streaming DDL: Time attribute (watermark) and Changelog support > > >> - Full SQL partition support for both batch & streaming [7] > > >> - New Java Expression DSL [8] > > >> - SQL CLI with DDL and DML support > > >> - Hive compatibility completion (DDL/UDF) to support full Hive > > integration > > >> - Partition/Function/View support > > >> - Remaining Blink planner/runtime merge > > >> - Support all TPC-DS queries [9] > > >> - Finer grained resource management > > >> - Unified TaskExecutor Memory Configuration [10] > > >> - Fine Grained Operator Resource Management [11] > > >> - Dynamic Slots Allocation [12] > > >> - Finish scheduler re-architecture [13] > > >> - Allows implementing more sophisticated scheduling strategies such > as > > >> better batch scheduler or speculative execution. > > >> - New DataStream Source Interface [14] > > >> - A new source connector architecture to unify the implementation of > > >> source connectors and make it simpler to implement custom source > > connectors. > > >> - Add more source/system metrics > > >> - For better flink job monitoring and facilitate customized > solutions > > >> like auto-scaling. > > >> - Executor Interface / Client API [15] > > >> - Allow Flink downstream projects to easier and better monitor and > > >> control flink jobs. > > >> - Interactive Programming [16] > > >> - Allow users to cache the intermediate results in Table API for > later > > >> usage to avoid redundant computation when a Flink application contains > > >> multiple jobs. > > >> - Python User Defined Function [17] > > >> - Support native user-defined functions in Flink Python, including > > >> UDF/UDAF/UDTF in Table API and Python-Java mixed UDF. > > >> - Spillable heap backend [18] > > >> - A new state backend supporting automatic data spill and load when > > >> memory exhausted/regained. > > >> - RocksDB backend memory control [19] > > >> - Prevent excessive memory usage from RocksDB, especially in > container > > >> environment. > > >> - Unaligned checkpoints [20] > > >> - Resolve the checkpoint timeout issue under backpressure. > > >> - Separate framework and user class loader in per-job mode > > >> - Active Kubernetes Integration [21] > > >> - Allow ResourceManager talking to Kubernetes to launch new pods > > >> similar to Flink's Yarn/Mesos integration > > >> - ML pipeline/library > > >> - Aims at delivering several core algorithms, including Logistic > > >> Regression, Native Bayes, Random Forest, KMeans, etc. > > >> - Add vertex subtask log url on WebUI [22] > > >> > > >> > > >> ** Suggested release timeline > > >> > > >> Based on our usual time-based release schedule [23], and considering > > that > > >> several events, such as Flink Forward Europe and Asia, are overlapping > > with > > >> the current release cycle, we should aim at releasing 1.10 around the > > >> beginning of January 2020. To give the community enough testing time, > I > > >> propose the feature freeze to be at the end of November. We should > > announce > > >> an > > >> exact date later in the release cycle. > > >> > > >> Lastly, I would like to use the opportunity to propose Yu Li and > myself > > as > > >> release managers for the upcoming release. > > >> > > >> What do you think? > > >> > > >> > > >> Best, > > >> Gary > > >> > > >> [1] > > >> > > > https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E > > >> [2] > > >> > > > https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E > > >> [3] https://issues.apache.org/jira/browse/FLINK-10725 > > >> [4] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration > > >> [5] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object > > >> [6] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > > >> [7] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > > >> [8] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL > > >> [9] https://issues.apache.org/jira/browse/FLINK-11491 > > >> [10] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > > >> [11] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > > >> [12] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > > >> [13] https://issues.apache.org/jira/browse/FLINK-10429 > > >> [14] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > > >> [15] > > >> > > > https://lists.apache.org/thread.html/498dd3e0277681cda356029582c1490299ae01df912e15942e11ae8e@%3Cdev.flink.apache.org%3E > > >> [16] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > > >> [17] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > > >> [18] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend > > >> [19] https://issues.apache.org/jira/browse/FLINK-7289 > > >> [20] > > >> > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html > > >> [21] > > >> > > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html > > >> [22] https://issues.apache.org/jira/browse/FLINK-13894 > > >> [23] > > https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > > >> > > > > > > > >
