Thanks Gary for kicking off the discussion. +1 for the feature freeze time. Also thanks Gary and Yu Li for volunteering as the release manager.
BTW, I'm working on refactoring of `CheckpointCoordinator` [1]. It would be great if it is included in 1.10. 1. https://issues.apache.org/jira/browse/FLINK-13698 Thanks, Biao /'bɪ.aʊ/ On Wed, 11 Sep 2019 at 18:33, Aljoscha Krettek <aljos...@apache.org> wrote: > Hi, > > Thanks for putting together the list! And I’m +1 for the suggested > release timeline and also for Gary and Yu as the release managers. > > Best, > Aljoscha > > On 9 Sep 2019, at 7:39, Yu Li wrote: > > > Hi Xuefu, > > > > If I understand it correctly, the data type support work should be > > included > > in the "Table API improvements->Finish type system" part, please check > > it > > and let us know if anything missing there. Thanks. > > > > Best Regards, > > Yu > > > > > > On Mon, 9 Sep 2019 at 11:14, Xuefu Z <usxu...@gmail.com> wrote: > > > >> Looking at feature list, I don't see an item for complete the data > >> type > >> support. Specifically, high precision timestamp is needed to Hive > >> integration, as it's so common. Missing it would damage the > >> completeness of > >> our Hive effort. > >> > >> Thanks, > >> Xuefu > >> > >> On Sat, Sep 7, 2019 at 7:06 PM Xintong Song <tonysong...@gmail.com> > >> wrote: > >> > >>> Thanks Gray and Yu for compiling the feature list and kicking off > >>> this > >>> discussion. > >>> > >>> +1 for Gary and Yu being the release managers for Flink 1.10. > >>> > >>> Thank you~ > >>> > >>> Xintong Song > >>> > >>> > >>> > >>> On Sat, Sep 7, 2019 at 4:58 PM Till Rohrmann <trohrm...@apache.org> > >> wrote: > >>> > >>>> Thanks for compiling the list of 1.10 efforts for the community > >>>> Gary. I > >>>> think this helps a lot to better understand what the community is > >>> currently > >>>> working on. > >>>> > >>>> Thanks for volunteering as the release managers for the next major > >>>> release. +1 for Gary and Yu being the RMs for Flink 1.10. > >>>> > >>>> Cheers, > >>>> Till > >>>> > >>>> On Sat, Sep 7, 2019 at 7:26 AM Zhu Zhu <reed...@gmail.com> wrote: > >>>> > >>>>> Thanks Gary for kicking off this discussion. > >>>>> Really appreciate that you and Yu offer to help to manage 1.10 > >> release. > >>>>> > >>>>> +1 for Gary and Yu as release managers. > >>>>> > >>>>> Thanks, > >>>>> Zhu Zhu > >>>>> > >>>>> Dian Fu <dian0511...@gmail.com> 于2019年9月7日周六 > >>>>> 下午12:26写道： > >>>>> > >>>>>> Hi Gary, > >>>>>> > >>>>>> Thanks for kicking off the release schedule of 1.10. +1 for you > >>>>>> and > >>> Yu > >>>> Li > >>>>>> as the release manager. > >>>>>> > >>>>>> The feature freeze/release time sounds reasonable. > >>>>>> > >>>>>> Thanks, > >>>>>> Dian > >>>>>> > >>>>>>> 在 2019年9月7日，上午11:30，Jark Wu <imj...@gmail.com> > >>>>>>> 写道： > >>>>>>> > >>>>>>> Thanks Gary for kicking off the discussion for 1.10 release. > >>>>>>> > >>>>>>> +1 for Gary and Yu as release managers. Thank you for you > >>>>>>> effort. > >>>>>>> > >>>>>>> Best, > >>>>>>> Jark > >>>>>>> > >>>>>>> > >>>>>>>> 在 2019年9月7日，00:52，zhijiang > >>>>>>>> <wangzhijiang...@aliyun.com.INVALID> > >>> 写道： > >>>>>>>> > >>>>>>>> Hi Gary, > >>>>>>>> > >>>>>>>> Thanks for kicking off the features for next release 1.10. I > >>>>>>>> am > >>>> very > >>>>>> supportive of you and Yu Li to be the relaese managers. > >>>>>>>> > >>>>>>>> Just mention another two improvements which want to be covered > >> in > >>>>>> FLINK-1.10 and I already confirmed with Piotr to reach an > >>>>>> agreement > >>>>> before. > >>>>>>>> > >>>>>>>> 1. Data serialize and copy only once for broadcast partition > >> [1]: > >>> It > >>>>>> would improve the throughput performance greatly in broadcast > >>>>>> mode > >>> and > >>>>> was > >>>>>> actually proposed in Flink-1.8. Most of works already done before > >> and > >>>>> only > >>>>>> left the last critical jira/PR. It will not take much efforts to > >> make > >>>> it > >>>>>> ready. > >>>>>>>> > >>>>>>>> 2. Let Netty use Flink's buffers directly in credit-based mode > >>> [2] : > >>>>> It > >>>>>> could avoid memory copy from netty stack to flink managed network > >>>> buffer. > >>>>>> The obvious benefit is decreasing the direct memory overhead > >> greatly > >>> in > >>>>>> large-scale jobs. I also heard of some user cases encounter > >>>>>> direct > >>> OOM > >>>>>> caused by netty memory overhead. Actually this improvment was > >>> proposed > >>>> by > >>>>>> nico in FLINK-1.7 and always no time to focus then. Yun Gao > >>>>>> already > >>>>>> submitted a PR half an year ago but have not been reviewed yet. I > >>> could > >>>>>> help review the deign and PR codes to make it ready. > >>>>>>>> > >>>>>>>> And you could make these two items as lowest priority if > >> possible. > >>>>>>>> > >>>>>>>> [1] https://issues.apache.org/jira/browse/FLINK-10745 > >>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-10742 > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Zhijiang > >>>>>>>> > >> ------------------------------------------------------------------ > >>>>>>>> From:Gary Yao <g...@apache.org> > >>>>>>>> Send Time:2019年9月6日(星期五) 17:06 > >>>>>>>> To:dev <dev@flink.apache.org> > >>>>>>>> Cc:carp84 <car...@gmail.com> > >>>>>>>> Subject:[DISCUSS] Features for Apache Flink 1.10 > >>>>>>>> > >>>>>>>> Hi community, > >>>>>>>> > >>>>>>>> Since Apache Flink 1.9.0 has been released more than 2 weeks > >> ago, > >>> I > >>>>>> want to > >>>>>>>> start kicking off the discussion about what we want to achieve > >> for > >>>> the > >>>>>> 1.10 > >>>>>>>> release. > >>>>>>>> > >>>>>>>> Based on discussions with various people as well as > >>>>>>>> observations > >>>> from > >>>>>>>> mailing > >>>>>>>> list threads, Yu Li and I have compiled a list of features that > >> we > >>>>> deem > >>>>>>>> important to be included in the next release. Note that the > >>> features > >>>>>>>> presented > >>>>>>>> here are not meant to be exhaustive. As always, I am sure that > >>> there > >>>>>> will be > >>>>>>>> other contributions that will make it into the next release. > >> This > >>>>> email > >>>>>>>> thread > >>>>>>>> is merely to kick off a discussion, and to give users and > >>>> contributors > >>>>>> an > >>>>>>>> understanding where the focus of the next release lies. If > >>>>>>>> there > >>> is > >>>>>> anything > >>>>>>>> we have missed that somebody is working on, please reply to > >>>>>>>> this > >>>>> thread. > >>>>>>>> > >>>>>>>> > >>>>>>>> ** Proposed features and focus > >>>>>>>> > >>>>>>>> Following the contribution of Blink to Apache Flink, the > >> community > >>>>>> released > >>>>>>>> a > >>>>>>>> preview of the Blink SQL Query Processor, which offers better > >> SQL > >>>>>> coverage > >>>>>>>> and > >>>>>>>> improved performance for batch queries, in Flink 1.9.0. > >>>>>>>> However, > >>> the > >>>>>>>> integration of the Blink query processor is not fully completed > >>> yet > >>>> as > >>>>>> there > >>>>>>>> are still pending tasks, such as implementing full TPC-DS > >> support. > >>>>> With > >>>>>> the > >>>>>>>> next Flink release, we aim at finishing the Blink integration. > >>>>>>>> > >>>>>>>> Furthermore, there are several ongoing work threads addressing > >>>>>> long-standing > >>>>>>>> issues reported by users, such as improving checkpointing under > >>>>>>>> backpressure, > >>>>>>>> and limiting RocksDBs native memory usage, which can be > >> especially > >>>>>>>> problematic > >>>>>>>> in containerized Flink deployments. > >>>>>>>> > >>>>>>>> Notable features surrounding Flink’s ecosystem that are > >>>>>>>> planned > >>> for > >>>>> the > >>>>>> next > >>>>>>>> release include active Kubernetes support (i.e., enabling > >> Flink’s > >>>>>>>> ResourceManager to launch new pods), improved Hive integration, > >>> Java > >>>>> 11 > >>>>>>>> support, and new algorithms for the Flink ML library. > >>>>>>>> > >>>>>>>> Below I have included the list of features that we compiled > >>> ordered > >>>> by > >>>>>>>> priority – some of which already have ongoing mailing list > >>> threads, > >>>>>> JIRAs, > >>>>>>>> or > >>>>>>>> FLIPs. > >>>>>>>> > >>>>>>>> - Improving Flink’s build system & CI [1] [2] > >>>>>>>> - Support Java 11 [3] > >>>>>>>> - Table API improvements > >>>>>>>> - Configuration Evolution [4] [5] > >>>>>>>> - Finish type system: Expression Re-design [6] and UDF > >> refactor > >>>>>>>> - Streaming DDL: Time attribute (watermark) and Changelog > >>> support > >>>>>>>> - Full SQL partition support for both batch & streaming [7] > >>>>>>>> - New Java Expression DSL [8] > >>>>>>>> - SQL CLI with DDL and DML support > >>>>>>>> - Hive compatibility completion (DDL/UDF) to support full Hive > >>>>>> integration > >>>>>>>> - Partition/Function/View support > >>>>>>>> - Remaining Blink planner/runtime merge > >>>>>>>> - Support all TPC-DS queries [9] > >>>>>>>> - Finer grained resource management > >>>>>>>> - Unified TaskExecutor Memory Configuration [10] > >>>>>>>> - Fine Grained Operator Resource Management [11] > >>>>>>>> - Dynamic Slots Allocation [12] > >>>>>>>> - Finish scheduler re-architecture [13] > >>>>>>>> - Allows implementing more sophisticated scheduling > >>>>>>>> strategies > >>>> such > >>>>> as > >>>>>>>> better batch scheduler or speculative execution. > >>>>>>>> - New DataStream Source Interface [14] > >>>>>>>> - A new source connector architecture to unify the > >>> implementation > >>>> of > >>>>>>>> source connectors and make it simpler to implement custom > >>>>>>>> source > >>>>>> connectors. > >>>>>>>> - Add more source/system metrics > >>>>>>>> - For better flink job monitoring and facilitate customized > >>>>> solutions > >>>>>>>> like auto-scaling. > >>>>>>>> - Executor Interface / Client API [15] > >>>>>>>> - Allow Flink downstream projects to easier and better > >>>>>>>> monitor > >>> and > >>>>>>>> control flink jobs. > >>>>>>>> - Interactive Programming [16] > >>>>>>>> - Allow users to cache the intermediate results in Table API > >> for > >>>>> later > >>>>>>>> usage to avoid redundant computation when a Flink application > >>>> contains > >>>>>>>> multiple jobs. > >>>>>>>> - Python User Defined Function [17] > >>>>>>>> - Support native user-defined functions in Flink Python, > >>> including > >>>>>>>> UDF/UDAF/UDTF in Table API and Python-Java mixed UDF. > >>>>>>>> - Spillable heap backend [18] > >>>>>>>> - A new state backend supporting automatic data spill and > >>>>>>>> load > >>>> when > >>>>>>>> memory exhausted/regained. > >>>>>>>> - RocksDB backend memory control [19] > >>>>>>>> - Prevent excessive memory usage from RocksDB, especially in > >>>>> container > >>>>>>>> environment. > >>>>>>>> - Unaligned checkpoints [20] > >>>>>>>> - Resolve the checkpoint timeout issue under backpressure. > >>>>>>>> - Separate framework and user class loader in per-job mode > >>>>>>>> - Active Kubernetes Integration [21] > >>>>>>>> - Allow ResourceManager talking to Kubernetes to launch new > >> pods > >>>>>>>> similar to Flink's Yarn/Mesos integration > >>>>>>>> - ML pipeline/library > >>>>>>>> - Aims at delivering several core algorithms, including > >> Logistic > >>>>>>>> Regression, Native Bayes, Random Forest, KMeans, etc. > >>>>>>>> - Add vertex subtask log url on WebUI [22] > >>>>>>>> > >>>>>>>> > >>>>>>>> ** Suggested release timeline > >>>>>>>> > >>>>>>>> Based on our usual time-based release schedule [23], and > >>> considering > >>>>>> that > >>>>>>>> several events, such as Flink Forward Europe and Asia, are > >>>> overlapping > >>>>>> with > >>>>>>>> the current release cycle, we should aim at releasing 1.10 > >> around > >>>> the > >>>>>>>> beginning of January 2020. To give the community enough testing > >>>> time, > >>>>> I > >>>>>>>> propose the feature freeze to be at the end of November. We > >> should > >>>>>> announce > >>>>>>>> an > >>>>>>>> exact date later in the release cycle. > >>>>>>>> > >>>>>>>> Lastly, I would like to use the opportunity to propose Yu Li > >>>>>>>> and > >>>>> myself > >>>>>> as > >>>>>>>> release managers for the upcoming release. > >>>>>>>> > >>>>>>>> What do you think? > >>>>>>>> > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Gary > >>>>>>>> > >>>>>>>> [1] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E > >>>>>>>> [2] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E > >>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-10725 > >>>>>>>> [4] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration > >>>>>>>> [5] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object > >>>>>>>> [6] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > >>>>>>>> [7] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > >>>>>>>> [8] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL > >>>>>>>> [9] https://issues.apache.org/jira/browse/FLINK-11491 > >>>>>>>> [10] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > >>>>>>>> [11] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > >>>>>>>> [12] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > >>>>>>>> [13] https://issues.apache.org/jira/browse/FLINK-10429 > >>>>>>>> [14] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > >>>>>>>> [15] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://lists.apache.org/thread.html/498dd3e0277681cda356029582c1490299ae01df912e15942e11ae8e@%3Cdev.flink.apache.org%3E > >>>>>>>> [16] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > >>>>>>>> [17] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > >>>>>>>> [18] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend > >>>>>>>> [19] https://issues.apache.org/jira/browse/FLINK-7289 > >>>>>>>> [20] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html > >>>>>>>> [21] > >>>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Best-practice-to-run-flink-on-kubernetes-td31532.html > >>>>>>>> [22] https://issues.apache.org/jira/browse/FLINK-13894 > >>>>>>>> [23] > >>>>>> > >>> https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > >> > >> -- > >> Xuefu Zhang > >> > >> "In Honey We Trust!" > >> >
