Hi，Zakelly Thank you for your answer. Best， rui
Zakelly Lan <zakelly....@gmail.com> 于2023年10月13日周五 19:12写道： > Hi rui, > > The 'state.backend.fs.memory-threshold' configures the threshold below > which state is stored as part of the metadata, rather than in separate > files. So as a result the JM will use its memory to merge small > checkpoint files and write them into one file. Currently the > FLIP-306[1][2] is proposed to merge small checkpoint files without > consuming JM memory. This feature is currently being worked on and is > targeted for the next minor release (1.19). > > > Best, > Zakelly > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-306%3A+Unified+File+Merging+Mechanism+for+Checkpoints > [2] https://issues.apache.org/jira/browse/FLINK-32070 > > On Fri, Oct 13, 2023 at 6:28 PM rui chen <ruichen...@gmail.com> wrote: > > > > We found that for some tasks, the JM memory continued to increase. I set > > the parameter of state.backend.fs.memory-threshold to 0, and the JM > memory > > would no longer increase, but many small files might be written in this > > way. Does the community have any optimization plan for this area? >
