Hi Stephan, Yes. I completely agree. Jincheng & Jark gave some very valuable feedbacks and suggestions and I think we can definitely move the conversation forward to reach a more concrete doc first before we put in to the roadmap. Thanks for reviewing it and driving the roadmap effort!
-- Rong On Thu, Feb 21, 2019 at 8:50 AM Stephan Ewen <se...@apache.org> wrote: > Hi Rong Rong! > > I would add the security / kerberos threads to the roadmap. They seem to > be advanced enough in the discussions so that there is clarity what will > come. > > For the window operator with slicing, I would personally like to see the > discussion advance and have some more clarity and consensus on the feature > before adding it to the roadmap. Not having that in the first version of > the roadmap does not mean there will be no activity. And when the > discussion advances well in the next weeks, we can update the roadmap soon. > > What do you think? > > Best, > Stephan > > > On Thu, Feb 14, 2019 at 5:46 PM Rong Rong <walter...@gmail.com> wrote: > >> Hi Stephan, >> >> Thanks for the clarification, yes I think these issues has already been >> discussed in previous mailing list threads [1,2,3]. >> >> I also agree that updating the "official" roadmap every release is a very >> good idea to avoid frequent update. >> One question I might've been a bit confusion is: are we suggesting to >> keep one roadmap on the documentation site (e.g. [4]) per release, or >> simply just one most up-to-date roadmap in the main website [5] ? >> Just like the release notes in every release, the former will probably >> provide a good tracker for users to look back at previous roadmaps as well >> I am assuming. >> >> Thanks, >> Rong >> >> [1] >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html >> [2] >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html >> [3] >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html >> >> [4] https://ci.apache.org/projects/flink/flink-docs-release-1.7/ >> [5] https://flink.apache.org/ >> >> On Thu, Feb 14, 2019 at 2:26 AM Stephan Ewen <se...@apache.org> wrote: >> >>> I think the website is better as well. >>> >>> I agree with Fabian that the wiki is not so visible, and visibility is >>> the main motivation. >>> This type of roadmap overview would not be updated by everyone - letting >>> committers update the roadmap means the listed threads are actually >>> happening at the moment. >>> >>> >>> On Thu, Feb 14, 2019 at 11:14 AM Fabian Hueske <fhue...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I like the idea of putting the roadmap on the website because it is >>>> much more visible (and IMO more credible, obligatory) there. >>>> However, I share the concerns about frequent updates. >>>> >>>> It think it would be great to update the "official" roadmap on the >>>> website once per release (-bugfix releases), i.e., every three month. >>>> We can use the wiki to collect and draft the roadmap for the next >>>> update. >>>> >>>> Best, Fabian >>>> >>>> >>>> Am Do., 14. Feb. 2019 um 11:03 Uhr schrieb Jeff Zhang <zjf...@gmail.com >>>> >: >>>> >>>>> Hi Stephan, >>>>> >>>>> Thanks for this proposal. It is a good idea to track the roadmap. One >>>>> suggestion is that it might be better to put it into wiki page first. >>>>> Because it is easier to update the roadmap on wiki compared to on flink >>>>> web >>>>> site. And I guess we may need to update the roadmap very often at the >>>>> beginning as there's so many discussions and proposals in community >>>>> recently. We can move it into flink web site later when we feel it could >>>>> be >>>>> nailed down. >>>>> >>>>> Stephan Ewen <se...@apache.org> 于2019年2月14日周四 下午5:44写道： >>>>> >>>>>> Thanks Jincheng and Rong Rong! >>>>>> >>>>>> I am not deciding a roadmap and making a call on what features should >>>>>> be developed or not. I was only collecting broader issues that are >>>>>> already >>>>>> happening or have an active FLIP/design discussion plus committer >>>>>> support. >>>>>> >>>>>> Do we have that for the suggested issues as well? If yes , we can add >>>>>> them (can you point me to the issue/mail-thread), if not, let's try and >>>>>> move the discussion forward and add them to the roadmap overview then. >>>>>> >>>>>> Best, >>>>>> Stephan >>>>>> >>>>>> >>>>>> On Wed, Feb 13, 2019 at 6:47 PM Rong Rong <walter...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Thanks Stephan for the great proposal. >>>>>>> >>>>>>> This would not only be beneficial for new users but also for >>>>>>> contributors to keep track on all upcoming features. >>>>>>> >>>>>>> I think that better window operator support can also be separately >>>>>>> group into its own category, as they affects both future DataStream API >>>>>>> and >>>>>>> batch stream unification. >>>>>>> can we also include: >>>>>>> - OVER aggregate for DataStream API separately as @jincheng >>>>>>> suggested. >>>>>>> - Improving sliding window operator [1] >>>>>>> >>>>>>> One more additional suggestion, can we also include a more >>>>>>> extendable security module [2,3] @shuyi and I are currently working on? >>>>>>> This will significantly improve the usability for Flink in corporate >>>>>>> environments where proprietary or 3rd-party security integration is >>>>>>> needed. >>>>>>> >>>>>>> Thanks, >>>>>>> Rong >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html >>>>>>> [2] >>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html >>>>>>> [3] >>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 13, 2019 at 3:39 AM jincheng sun < >>>>>>> sunjincheng...@gmail.com> wrote: >>>>>>> >>>>>>>> Very excited and thank you for launching such a great discussion, >>>>>>>> Stephan ! >>>>>>>> >>>>>>>> Here only a little suggestion that in the Batch Streaming >>>>>>>> Unification section, do we need to add an item: >>>>>>>> >>>>>>>> - Same window operators on bounded/unbounded Table API and >>>>>>>> DataStream API >>>>>>>> (currently OVER window only exists in SQL/TableAPI, DataStream API >>>>>>>> does not yet support) >>>>>>>> >>>>>>>> Best, >>>>>>>> Jincheng >>>>>>>> >>>>>>>> Stephan Ewen <se...@apache.org> 于2019年2月13日周三 下午7:21写道： >>>>>>>> >>>>>>>>> Hi all! >>>>>>>>> >>>>>>>>> Recently several contributors, committers, and users asked about >>>>>>>>> making it more visible in which way the project is currently going. >>>>>>>>> >>>>>>>>> Users and developers can track the direction by following the >>>>>>>>> discussion threads and JIRA, but due to the mass of discussions and >>>>>>>>> open >>>>>>>>> issues, it is very hard to get a good overall picture. >>>>>>>>> Especially for new users and contributors, is is very hard to get >>>>>>>>> a quick overview of the project direction. >>>>>>>>> >>>>>>>>> To fix this, I suggest to add a brief roadmap summary to the >>>>>>>>> homepage. It is a bit of a commitment to keep that roadmap up to >>>>>>>>> date, but >>>>>>>>> I think the benefit for users justifies that. >>>>>>>>> The Apache Beam project has added such a roadmap [1] >>>>>>>>> <https://beam.apache.org/roadmap/>, which was received very well >>>>>>>>> by the community, I would suggest to follow a similar structure here. >>>>>>>>> >>>>>>>>> If the community is in favor of this, I would volunteer to write a >>>>>>>>> first version of such a roadmap. The points I would include are below. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Stephan >>>>>>>>> >>>>>>>>> [1] https://beam.apache.org/roadmap/ >>>>>>>>> >>>>>>>>> ======================================================== >>>>>>>>> >>>>>>>>> Disclaimer: Apache Flink is not governed or steered by any one >>>>>>>>> single entity, but by its community and Project Management Committee >>>>>>>>> (PMC). >>>>>>>>> This is not a authoritative roadmap in the sense of a plan with a >>>>>>>>> specific >>>>>>>>> timeline. Instead, we share our vision for the future and major >>>>>>>>> initiatives >>>>>>>>> that are receiving attention and give users and contributors an >>>>>>>>> understanding what they can look forward to. >>>>>>>>> >>>>>>>>> *Future Role of Table API and DataStream API* >>>>>>>>> - Table API becomes first class citizen >>>>>>>>> - Table API becomes primary API for analytics use cases >>>>>>>>> * Declarative, automatic optimizations >>>>>>>>> * No manual control over state and timers >>>>>>>>> - DataStream API becomes primary API for applications and data >>>>>>>>> pipeline use cases >>>>>>>>> * Physical, user controls data types, no magic or optimizer >>>>>>>>> * Explicit control over state and time >>>>>>>>> >>>>>>>>> *Batch Streaming Unification* >>>>>>>>> - Table API unification (environments) (FLIP-32) >>>>>>>>> - New unified source interface (FLIP-27) >>>>>>>>> - Runtime operator unification & code reuse between DataStream / >>>>>>>>> Table >>>>>>>>> - Extending Table API to make it convenient API for all >>>>>>>>> analytical use cases (easier mix in of UDFs) >>>>>>>>> - Same join operators on bounded/unbounded Table API and >>>>>>>>> DataStream API >>>>>>>>> >>>>>>>>> *Faster Batch (Bounded Streams)* >>>>>>>>> - Much of this comes via Blink contribution/merging >>>>>>>>> - Fine-grained Fault Tolerance on bounded data (Table API) >>>>>>>>> - Batch Scheduling on bounded data (Table API) >>>>>>>>> - External Shuffle Services Support on bounded streams >>>>>>>>> - Caching of intermediate results on bounded data (Table API) >>>>>>>>> - Extending DataStream API to explicitly model bounded streams >>>>>>>>> (API breaking) >>>>>>>>> - Add fine fault tolerance, scheduling, caching also to >>>>>>>>> DataStream API >>>>>>>>> >>>>>>>>> *Streaming State Evolution* >>>>>>>>> - Let all built-in serializers support stable evolution >>>>>>>>> - First class support for other evolvable formats (Protobuf, >>>>>>>>> Thrift) >>>>>>>>> - Savepoint input/output format to modify / adjust savepoints >>>>>>>>> >>>>>>>>> *Simpler Event Time Handling* >>>>>>>>> - Event Time Alignment in Sources >>>>>>>>> - Simpler out-of-the box support in sources >>>>>>>>> >>>>>>>>> *Checkpointing* >>>>>>>>> - Consistency of Side Effects: suspend / end with savepoint >>>>>>>>> (FLIP-34) >>>>>>>>> - Failed checkpoints explicitly aborted on TaskManagers (not >>>>>>>>> only on coordinator) >>>>>>>>> >>>>>>>>> *Automatic scaling (adjusting parallelism)* >>>>>>>>> - Reactive scaling >>>>>>>>> - Active scaling policies >>>>>>>>> >>>>>>>>> *Kubernetes Integration* >>>>>>>>> - Active Kubernetes Integration (Flink actively manages >>>>>>>>> containers) >>>>>>>>> >>>>>>>>> *SQL Ecosystem* >>>>>>>>> - Extended Metadata Stores / Catalog / Schema Registries support >>>>>>>>> - DDL support >>>>>>>>> - Integration with Hive Ecosystem >>>>>>>>> >>>>>>>>> *Simpler Handling of Dependencies* >>>>>>>>> - Scala in the APIs, but not in the core (hide in separate class >>>>>>>>> loader) >>>>>>>>> - Hadoop-free by default >>>>>>>>> >>>>>>>>> >>>>> >>>>> -- >>>>> Best Regards >>>>> >>>>> Jeff Zhang >>>>> >>>>
