Hi Zakelly, Thanks for driving this, this proposal enables the files merging of different types of states to be grouped under a unified framework. I think it has the added benefit of lightening the load on JM. As FLINK-26590[1] described, triggered checkpoints can be delayed by discarding shared state when JM manages a large number of files. After this FLIP, JM only needs to manage some folders, which greatly reduces the burden on JM.
In Section 4.1, two types of merging granularities(per subtask and per task manager) are proposed, the shared state is managed by per subtask granularity, but for the changelog state backend, its DSTL files are shared between checkpoints, and are currently merged in batches at the task manager level. When merging with the SEGMENTED_WITHIN_CP_BOUNDARY mode, I'm concerned about the performance degradation of its merging, hence I wonder if the merge granularities are configurable? Further, from a user perspective, three new options are introduced in this FLIP, do they have recommended defaults? [1] https://issues.apache.org/jira/browse/FLINK-26590 Best, Yanfei Zakelly Lan <zakelly....@gmail.com> 于2023年4月3日周一 18:36写道： > > Hi everyone, > > I would like to open a discussion on providing a unified file merging > mechanism for checkpoints[1]. > > Currently, many files are uploaded to the DFS during checkpoints, > leading to the 'file flood' problem when running > intensive workloads in a cluster. To tackle this problem, various > solutions have been proposed for different types > of state files. Although these methods are similar, they lack a > systematic view and approach. We believe that it is > better to consider this problem as a whole and introduce a unified > framework to address the file flood problem for > all types of state files. A POC has been implemented based on current > FLIP design, and the test results are promising. > > > Looking forward to your comments or feedback. > > Best regards, > Zakelly > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints
