Hi, Weijie Thanks for your feedback!
`StreamGraphOptimizationStrategy` is a reasonable abstract, I'd like to know what built-in strategy implementations you have in mind so far? We will introduce two optimization strategies: AdaptiveBroadcastJoinOptimizeStrategy, which dynamically determines and switches to BroadcastJoin, and SkewedJoinOptimizeStrategy, which addresses data skew issues. For the so-called pending operators, can we show it in different colors on the UI. Yes, we will use different colors (such as green) to display the pending operators. Best regards, Junrui weijie guo <guoweijieres...@gmail.com> 于2024年7月17日周三 20:15写道： > Thanks for the proposal! > > I like this idea as it gives Flink's adaptive batching processing more room > to imagine and optimize. > > So, +1 from my side. > > I just have two questions: > > 1. `StreamGraphOptimizationStrategy` is a reasonable abstract, I'd like to > know what built-in strategy implementations you have in mind so far? > > 2. For the so-called pending operators, can we show it in different colors > on the UI. > > > Best regards, > > Weijie > > > Zhu Zhu <reed...@gmail.com> 于2024年7月17日周三 16:49写道： > > > Thanks Junrui for the updates. The proposal looks good to me. > > With the stream graph added to the REST API result, I think we are > > also quite close to enable Flink to expand a job vertex to show its > > operator-chain topology. > > > > Thanks, > > Zhu > > > > Junrui Lee <jrlee....@gmail.com> 于2024年7月15日周一 14:58写道： > > > > > Hi Zhu, > > > > > > Thanks for your feedback. > > > > > > Following your suggestion, I have updated the public interface section > of > > > the FLIP with the following additions: > > > > > > 1. UI: > > > The job topology will display a hybrid of the current JobGraph along > with > > > downstream components yet to be converted to a StreamGraph. On the > > topology > > > graph display page, there will be a "Show Pending Operators" button in > > the > > > upper right corner for users to switch back to a job topology that only > > > includes JobVertices. > > > > > > 2. Rest API: > > > Add a new field "stream-graph-plan" will be added to the job details > REST > > > API, which represents the runtime Stream graph. The field > "job-vertex-id" > > > is valid only when the StreamNode has been converted to a JobVertex, > and > > it > > > will hold the ID of the corresponding JobVertex for that StreamNode. > > > > > > For further information, please feel free to review the public > interface > > > section of FLIP-469 > > > > > > Best, > > > Junrui > > > > > > Zhu Zhu <reed...@gmail.com> 于2024年7月15日周一 10:29写道： > > > > > > > +1 for the FLIP > > > > > > > > It is useful to adaptively optimize logical execution plans(stream > > > > operators and > > > > stream edges) for batch jobs. > > > > > > > > One question: > > > > The FLIP already proposed to update the REST API & Web UI to show > > > operators > > > > that are not yet converted to job vertices. However, I think it would > > be > > > > better if Flink can display these operators as part of the graph, > > > allowing > > > > users to have an overview of the processing logic graph at early > stages > > > of > > > > the job execution. > > > > This change would also involve the public interface, so instead of > > > > postponing > > > > it to a later FLIP, I prefer to have a design for it in this FLIP. > > WDYT? > > > > > > > > Thanks, > > > > Zhu > > > > > > > > Junrui Lee <jrlee....@gmail.com> 于2024年7月11日周四 11:27写道： > > > > > > > > > Hi devs, > > > > > > > > > > Xia Sun, Lei Yang, and I would like to initiate a discussion about > > > > > FLIP-469: Supports Adaptive Optimization of StreamGraph. > > > > > > > > > > This FLIP is the second in the series on adaptive optimization of > > > > > StreamGraph and follows up on FLIP-468 [1]. As we proposed in > > FLIP-468 > > > to > > > > > enable the scheduler to recognize and access the StreamGraph, in > this > > > > FLIP, > > > > > we will propose a mechanism for adaptive optimization of > StreamGraph. > > > It > > > > > allows the scheduler to dynamically adjust the logical execution > plan > > > at > > > > > runtime. This mechanism is the base of adaptive optimization > > > strategies, > > > > > such as adaptive broadcast join and skewed join optimization. > > > > > > > > > > Unlike the traditional execution of jobs based on a static > > StreamGraph, > > > > > this mechanism will progressively determine StreamGraph during > > runtime. > > > > The > > > > > determined StreamGraph will be transformed into a specific > JobGraph, > > > > while > > > > > the indeterminate part will allow Flink to flexibly adjust > according > > to > > > > > real-time job status and actual input conditions. > > > > > > > > > > Note that this FLIP focuses on the introduction of the mechanism > and > > > does > > > > > not introduce any actual optimization strategies; these will be > > > discussed > > > > > in subsequent FLIPs. > > > > > > > > > > For more details, please refer to FLIP-469 [2]. We look forward to > > your > > > > > feedback. > > > > > > > > > > Best, > > > > > > > > > > Xia Sun, Lei Yang and Junrui Lee > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-469%3A+Supports+Adaptive+Optimization+of+StreamGraph > > > > > > > > > > > > > > >
