+1 for the FLIP. With Dataset API removed in Flink 2.0, StreamGraph will be the only operator level DAG of a Flink job.
Exposing it directly to the scheduler can be helpful to deliver some exciting features. This FLIP meationed adaptive plan optimization and operator insights. I'm also thinking of features like dynamic chaining breakdown for perf-tuning and diagnostic. Thanks, Zhu Junrui Lee <jrlee....@gmail.com> 于2024年7月11日周四 11:25写道： > Hi devs, > > Xia Sun, Lei Yang, and I would like to initiate a discussion about > FLIP-468: Introducing StreamGraph-Based Job Submission. > > Currently, Flink has the capability to adjust the ExecutionGraph in batch > processing mode, such as dynamically deciding the parallelism of JobVertex > based on the input data. However, in certain scenarios, adjustments to the > ExecutionGraph are insufficient to resolve issues. At this point, we need > to adjust the StreamGraph, including the logic of operators and the data > distribution patterns. For instance, adaptively changing hash join or sort > merge join to broadcast join when the data volume of one input to a Join > operator is small and the other input is large can improve performance. > Meanwhile, re-partitioning or adjusting the computational logic can be > beneficial to solve data hotspots. > > As a result, we intend to introduce a mechanism for adaptive optimization > of StreamGraph, and leverage it to support adaptive broadcast join and > skewed join optimization. > > This FLIP is the first in a series, aiming to introduce a job submission > mode based on StreamGraph. This mode will enable Flink to directly access > and adjust the actual logical execution plan of the job at runtime, to > enhance the job's execution performance and observability. > > For more details, please refer to FLIP-468 [1]. We look forward to your > feedback. > > Best, > > Xia Sun, Lei Yang and Junrui Lee > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission >
