David, Junrui, Is there a brief description of the prerequisites for compiled plan-based submission?
Cheers, Jim On Fri, Aug 9, 2024 at 3:55 AM David Morávek <d...@apache.org> wrote: > Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed offline > conversation. It was incredibly helpful in aligning on our north star for > job submission. > > We’ve agreed that FLIP-468 serves as a temporary solution with acceptable > risks regarding API stability, which are outweighed by the new features and > opportunities it brings. Our long-term goal remains to transition to a > compiled plan-based submission once the necessary prerequisites are in > place. > > Let’s proceed with the voting. I’m giving a +1 from my side. > > Thanks for bearing with me. > > Best, > D. > > > On Tue, Aug 6, 2024 at 4:04 PM Junrui Lee <jrlee....@gmail.com> wrote: > > > Hi everyone, > > > > David, Zhu and I had an offline discussion about FLIP-468 and more. Here > > are the key points: > > 1. Flink lacks a stable public REST API for job submission. Such a public > > REST API will allow users to create and store compiled plans of jobs, and > > to directly submit/resumit these plans. It can be helpful for making job > > plans more stable across versions, speeding up job submission, and > > supporting version-less job submission clients. > > 2. FLIP-468 mainly focuses on how to enable JobMaster/Scheduler to see > and > > understand StreamGraph. It is applicable for both SQL and DataStream > jobs. > > This FLIP will modify the existing jobgraph submission REST API. But that > > is an internal API for Flink clients and the change will not affect > users. > > Therefore, it does not conflict with the target of #1. > > > > The conclusion is: > > 1. We will continue with FLIP-468 as is. > > 2. Later we will consider how to introduce a pubic REST API for job > > submission. This will happen in another FLIP. > > > > Thanks David and Zhu for the inputs and thoughts. > > As the content of FLIP-468 is not changed, we will continue the voting > > using the previous voting ML[1]. > > > > [1] https://lists.apache.org/thread/d7md9bprfhynjfq7vy8dqld2dn9lf0ob > > > > Best, > > Junrui > > > > David Morávek <d...@apache.org> 于 2024年7月29日周一 16:53写道： > > > > > Hi everyone, > > > > > > I hope this message finds you well. I recently returned from a two-week > > > vacation and apologize for the delay in my response. > > > > > > There are still a few crucial questions, especially regarding how this > > will > > > work with streaming that need to be addressed. Therefore, I would like > to > > > temporarily cast a -1 vote until these outstanding issues are resolved. > > > I'll provide more detailed in the discussion thread. > > > > > > Thank you for your understanding. > > > > > > Best regards, > > > D. > > > > > > On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen < > zhanghao.c...@outlook.com > > > > > > wrote: > > > > > > > Thanks for driving it. +1 (non-binding) > > > > > > > > Best, > > > > Zhanghao Chen > > > > ________________________________ > > > > From: Junrui Lee <jrlee....@gmail.com> > > > > Sent: Friday, July 26, 2024 11:02 > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job > Submission > > > > > > > > Hi everyone, > > > > > > > > Thanks for all the feedback about FLIP-468: Introducing > > StreamGraph-Based > > > > Job Submission [1]. The discussion thread can be found here [2]. > > > > > > > > The vote will be open for at least 72 hours unless there are any > > > objections > > > > or insufficient votes. > > > > > > > > Best, > > > > > > > > Junrui > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission > > > > > > > > [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz > > > > > > > > > >
