tldr, compiled plan needs to be stable and feature complete; covering sql streaming, batch and datastream api
On Fri 9. 8. 2024 at 15:58, Jim Hughes <jhug...@confluent.io.invalid> wrote: > David, Junrui, > > Is there a brief description of the prerequisites for compiled plan-based > submission? > > Cheers, > > Jim > > On Fri, Aug 9, 2024 at 3:55 AM David Morávek <d...@apache.org> wrote: > > > Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed > offline > > conversation. It was incredibly helpful in aligning on our north star for > > job submission. > > > > We’ve agreed that FLIP-468 serves as a temporary solution with acceptable > > risks regarding API stability, which are outweighed by the new features > and > > opportunities it brings. Our long-term goal remains to transition to a > > compiled plan-based submission once the necessary prerequisites are in > > place. > > > > Let’s proceed with the voting. I’m giving a +1 from my side. > > > > Thanks for bearing with me. > > > > Best, > > D. > > > > > > On Tue, Aug 6, 2024 at 4:04 PM Junrui Lee <jrlee....@gmail.com> wrote: > > > > > Hi everyone, > > > > > > David, Zhu and I had an offline discussion about FLIP-468 and more. > Here > > > are the key points: > > > 1. Flink lacks a stable public REST API for job submission. Such a > public > > > REST API will allow users to create and store compiled plans of jobs, > and > > > to directly submit/resumit these plans. It can be helpful for making > job > > > plans more stable across versions, speeding up job submission, and > > > supporting version-less job submission clients. > > > 2. FLIP-468 mainly focuses on how to enable JobMaster/Scheduler to see > > and > > > understand StreamGraph. It is applicable for both SQL and DataStream > > jobs. > > > This FLIP will modify the existing jobgraph submission REST API. But > that > > > is an internal API for Flink clients and the change will not affect > > users. > > > Therefore, it does not conflict with the target of #1. > > > > > > The conclusion is: > > > 1. We will continue with FLIP-468 as is. > > > 2. Later we will consider how to introduce a pubic REST API for job > > > submission. This will happen in another FLIP. > > > > > > Thanks David and Zhu for the inputs and thoughts. > > > As the content of FLIP-468 is not changed, we will continue the voting > > > using the previous voting ML[1]. > > > > > > [1] https://lists.apache.org/thread/d7md9bprfhynjfq7vy8dqld2dn9lf0ob > > > > > > Best, > > > Junrui > > > > > > David Morávek <d...@apache.org> 于 2024年7月29日周一 16:53写道： > > > > > > > Hi everyone, > > > > > > > > I hope this message finds you well. I recently returned from a > two-week > > > > vacation and apologize for the delay in my response. > > > > > > > > There are still a few crucial questions, especially regarding how > this > > > will > > > > work with streaming that need to be addressed. Therefore, I would > like > > to > > > > temporarily cast a -1 vote until these outstanding issues are > resolved. > > > > I'll provide more detailed in the discussion thread. > > > > > > > > Thank you for your understanding. > > > > > > > > Best regards, > > > > D. > > > > > > > > On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen < > > zhanghao.c...@outlook.com > > > > > > > > wrote: > > > > > > > > > Thanks for driving it. +1 (non-binding) > > > > > > > > > > Best, > > > > > Zhanghao Chen > > > > > ________________________________ > > > > > From: Junrui Lee <jrlee....@gmail.com> > > > > > Sent: Friday, July 26, 2024 11:02 > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > > Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job > > Submission > > > > > > > > > > Hi everyone, > > > > > > > > > > Thanks for all the feedback about FLIP-468: Introducing > > > StreamGraph-Based > > > > > Job Submission [1]. The discussion thread can be found here [2]. > > > > > > > > > > The vote will be open for at least 72 hours unless there are any > > > > objections > > > > > or insufficient votes. > > > > > > > > > > Best, > > > > > > > > > > Junrui > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission > > > > > > > > > > [2] > https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz > > > > > > > > > > > > > > >
