[
https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729143#comment-17729143
]
Rui Fan commented on FLINK-26803:
---------------------------------
{quote}The PR of this JIRA introduced a new public evolving API
CheckpointConfig#getMaxSubtasksPerChannelStateFile and public config
execution.checkpointing.unaligned.max-subtasks-per-channel-state-file. I could
not find the FLIP associated with this JIRA.
Is there a FLIP for this change?
{quote}
Hi [~lindong] , sorry, this JIRA doesn't have FLIP, it just have a design
doc[1].
{quote}FLIP-306 seems to be relate to this JIRA. But the newly added config
execution.checkpointing.unaligned.max-subtasks-per-channel-state-file is not
found in FLIP-306.
{quote}
Actually, this JIRA is related to FLIP-306
* FLIP-306 targets various checkpoint file merging mechanisms.
* This JIRA has been finished at flink 1.17. It was proposed earlier than
FLIP-306, and only for the merge mechanism of channel state files.
So it's not a subtask of FLIP-306. Of course, I think FLIP-306 is more general
for file merging, looking forward to the completion of FLIP-306.
[1]
[https://docs.google.com/document/d/1uGp1gJxxcJMUCh5DxzPklGc2SicMN-Dk-YOce2XMaAg/edit?usp=sharing]
> Merge small ChannelState file for Unaligned Checkpoint
> ------------------------------------------------------
>
> Key: FLINK-26803
> URL: https://issues.apache.org/jira/browse/FLINK-26803
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Network
> Reporter: Rui Fan
> Assignee: Rui Fan
> Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.17.0
>
> Attachments: image-2022-05-05-12-36-09-969.png
>
>
> When making an unaligned checkpoint, the number of ChannelState files is
> TaskNumber * subtaskNumber. For high parallelism job, it writes too many
> small files. It causes high load for hdfs NN.
>
> In our production, a job writes more than 50K small files for each Unaligned
> Checkpoint. Could we merge these files before write FileSystem? We can
> configure the maximum number of files each TM can write in a single Unaligned
> Checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
