Thanks Junrui for the updates. The proposal looks good to me. With the stream graph added to the REST API result, I think we are also quite close to enable Flink to expand a job vertex to show its operator-chain topology.
Thanks, Zhu Junrui Lee <jrlee....@gmail.com> 于2024年7月15日周一 14:58写道： > Hi Zhu, > > Thanks for your feedback. > > Following your suggestion, I have updated the public interface section of > the FLIP with the following additions: > > 1. UI: > The job topology will display a hybrid of the current JobGraph along with > downstream components yet to be converted to a StreamGraph. On the topology > graph display page, there will be a "Show Pending Operators" button in the > upper right corner for users to switch back to a job topology that only > includes JobVertices. > > 2. Rest API: > Add a new field "stream-graph-plan" will be added to the job details REST > API, which represents the runtime Stream graph. The field "job-vertex-id" > is valid only when the StreamNode has been converted to a JobVertex, and it > will hold the ID of the corresponding JobVertex for that StreamNode. > > For further information, please feel free to review the public interface > section of FLIP-469 > > Best, > Junrui > > Zhu Zhu <reed...@gmail.com> 于2024年7月15日周一 10:29写道： > > > +1 for the FLIP > > > > It is useful to adaptively optimize logical execution plans(stream > > operators and > > stream edges) for batch jobs. > > > > One question: > > The FLIP already proposed to update the REST API & Web UI to show > operators > > that are not yet converted to job vertices. However, I think it would be > > better if Flink can display these operators as part of the graph, > allowing > > users to have an overview of the processing logic graph at early stages > of > > the job execution. > > This change would also involve the public interface, so instead of > > postponing > > it to a later FLIP, I prefer to have a design for it in this FLIP. WDYT? > > > > Thanks, > > Zhu > > > > Junrui Lee <jrlee....@gmail.com> 于2024年7月11日周四 11:27写道： > > > > > Hi devs, > > > > > > Xia Sun, Lei Yang, and I would like to initiate a discussion about > > > FLIP-469: Supports Adaptive Optimization of StreamGraph. > > > > > > This FLIP is the second in the series on adaptive optimization of > > > StreamGraph and follows up on FLIP-468 [1]. As we proposed in FLIP-468 > to > > > enable the scheduler to recognize and access the StreamGraph, in this > > FLIP, > > > we will propose a mechanism for adaptive optimization of StreamGraph. > It > > > allows the scheduler to dynamically adjust the logical execution plan > at > > > runtime. This mechanism is the base of adaptive optimization > strategies, > > > such as adaptive broadcast join and skewed join optimization. > > > > > > Unlike the traditional execution of jobs based on a static StreamGraph, > > > this mechanism will progressively determine StreamGraph during runtime. > > The > > > determined StreamGraph will be transformed into a specific JobGraph, > > while > > > the indeterminate part will allow Flink to flexibly adjust according to > > > real-time job status and actual input conditions. > > > > > > Note that this FLIP focuses on the introduction of the mechanism and > does > > > not introduce any actual optimization strategies; these will be > discussed > > > in subsequent FLIPs. > > > > > > For more details, please refer to FLIP-469 [2]. We look forward to your > > > feedback. > > > > > > Best, > > > > > > Xia Sun, Lei Yang and Junrui Lee > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission > > > [2] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-469%3A+Supports+Adaptive+Optimization+of+StreamGraph > > > > > >
