Thanks Yu and Gary for the detailed summary and update! On Fri, Sep 27, 2019 at 6:54 AM Yu Li <car...@gmail.com> wrote:
> Hi community, > > Since we are now more than one month into the Flink 1.10 release cycle, we > thought it would be adequate to give a progress update. Below we have > included a list of the ongoing efforts that we are aware of, together with > a brief summary of their state. As always, the list is not meant to be > exhaustive. If you are working on something that is not included here, feel > free to use this thread to share your progress. > > Note that because we are still relatively at the beginning of the release > cycle, most of the progress is limited to FLIPs that are accepted or being > voted on. > > - Improving Flink’s build system & CI > - Repository Split [1] > - Discussed on the ML but consensus to split the repository was not > reached. > - Reduce Build Time [2] > - Discussion is ongoing. Currently, using Azure Pipelines and > Gradle are being evaluated. > > - Support Java 11 [3] > - Implementation is in progress (18/21 subtasks resolved) > > - Table API improvements > - FLIP-54 Evolve ConfigOption and Configuration [4] > - Under discussion. > - FLIP-59 Enable Execution Configuration from Configuration Object [5] > - Under discussion. > - Full Data Type Support in Planner [6] > - Implementation in progress. > - FLIP-66 Support Time Attribute in SQL DDL [7] > - FLIP voting. > - FLIP-70 Support Computed Column [8] > - Under discussion. > - FLIP-63 Rework Table Partition Support [9] > - FLIP voting > - FLIP-51 Rework of Expression Design [10] > - FLIP accepted, implementation in progress. > - FLIP-55 Introduction of a TableAPI Java Expression DSL [11] > - Under discussion. > - FLIP-64 Support for Temporary Objects in Table Module [12] > - Under discussion. > - FLIP-65 New Type Inference for Table API UDFs > - Under discussion. > > - Hive compatibility completion (DDL/UDF) to support full Hive integration > - FLIP-57 Rework FunctionCatalog [13] > - FLIP voting > - FLIP-68 Extend Core Table System with Modular Plugins [14] > - FLIP voting was initiated [15] but temporarily withdrawn due to > lack of community bandwidth. > > - Finer grained resource management > - FLIP-49: Unified Memory Configuration for TaskExecutors [16] > - FLIP accepted. Implementation is in progress. > - FLIP-53: Fine Grained Operator Resource Management [17] > - FLIP accepted. Implementation details are under discussion. > - FLIP-56: Dynamic Slot Allocation [18] > - FLIP accepted. Implementation not started yet. > > - Finish scheduler re-architecture [19] > - Implementation is in progress. > > - FLIP-27: Refactor Source Interface [20] > - FLIP accepted. Implementation is in progress. > > - Executor/Client refactoring [21] > - Discussion already reached consensus > - FLIP is coming. A PoC implementation is also ready. > > - FLIP-36 Support Interactive Programming [22] > - Reviewing FLIP-67, which changes the intermediate result management > in runtime, which is what FLIP-36 will be built on top of. > > - FLIP-58: Flink Python User-Defined Stateless Function for Table [23] > - Implementation is in progress (3/15 subtask resolved). > - Python environment and dependency management under discussion > > - FLIP-50: Spill-able Heap Keyed State Backend [24] > - FLIP was accepted. Implementation is in progress. > > - RocksDB Backend Memory Control [25] > - Verified capping memory usage through Write Buffer Manager [26] works > in production. > - New RocksDB version TBD, 5.18.3/6.2.2 has performance regression [27] > compared to the currently used version 5.17.2. > - FLIP of MemoryManager interface for reserving memory to be opened. > > - Unaligned Checkpoints [28] > - Design under discussion. > - FLIP document is under development and will be released shortly > > - Separate framework and user class loader in per-job mode [29] > - Pull request is being reviewed. > > - Active Kubernetes Integration [30] > - PoC completed. More details need to be discussed before updating the > PRs. > > - FLIP-39 Flink ML pipeline and ML libs [31] > - ML pipeline API PRs (FLINK-13339) have been opened and are being > reviewed. > - Algorithms are waiting for the new ML pipeline API to be merged. > > - Add vertex subtask log url on WebUI [32] > - This makes it easier for users of the WebUI to access the logs of the > TaskManager that executes a specific subtask. > - A pull request is opened and currently being reviewed. > > As a reminder, the feature freeze is targeted to be at the end of November. > This leaves us with approximately another 2 months of development time. We > will send another announcement later in the release cycle to make the date > of the feature freeze official. > > Best, > Gary & Yu > > [1] > > https://lists.apache.org/thread.html/775447a187410727f5ba6f9cefd6406c58ca5cc5c580aecf30cf213e@%3Cdev.flink.apache.org%3E > [2] > > https://lists.apache.org/thread.html/b90aa518fcabce94f8e1de4132f46120fae613db6e95a2705f1bd1ea@%3Cdev.flink.apache.org%3E > [3] https://issues.apache.org/jira/browse/FLINK-10725 > [4] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-54%3A+Evolve+ConfigOption+and+Configuration > [5] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-59%3A+Enable+execution+configuration+from+Configuration+object > [6] https://issues.apache.org/jira/browse/FLINK-14079 > [7] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-66%3A+Support+time+attribute+in+SQL+DDL > [8] > > https://docs.google.com/document/d/110TseRtTCphxETPY7uhiHpu-dph3NEesh3mYKtJ7QOY/edit?usp=sharing > [9] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-63%3A+Rework+table+partition+support > [10] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design > [11] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-55%3A+Introduction+of+a+Table+API+Java+Expression+DSL > [12] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module > [13] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-57%3A+Rework+FunctionCatalog > [14] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-68%3A+Extend+Core+Table+System+with+Modular+Plugins > [15] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-68-Extend-Core-Table-System-with-Modular-Plugins-td33372.html > [16] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors > [17] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management > [18] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation > [19] https://issues.apache.org/jira/browse/FLINK-10429 > [20] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > [21] > > https://lists.apache.org/thread.html/ce99cba4a10b9dc40eb729d39910f315ae41d80ec74f09a356c73938@%3Cdev.flink.apache.org%3E > [22] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > [23] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-58%3A+Flink+Python+User-Defined+Stateless+Function+for+Table > [24] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-50%3A+Spill-able+Heap+Keyed+State+Backend > [25] https://issues.apache.org/jira/browse/FLINK-7289 > [26] https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager > [27] https://github.com/facebook/rocksdb/issues/5774 > [28] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Checkpointing-under-backpressure-td31616.html > [29] https://issues.apache.org/jira/browse/FLINK-13993 > [30] https://issues.apache.org/jira/browse/FLINK-9953 > [31] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs > [32] https://issues.apache.org/jira/browse/FLINK-13894 >
