Hi，Zakelly
Thank you for your answer.
Best，
rui
Zakelly Lan 于2023年10月13日周五 19:12写道：
> Hi rui,
>
> The 'state.backend.fs.memory-threshold' configures the threshold below
> which state is stored as part of the metadata, rather than in separate
> files. So as a result the JM will use its memory t
Hi rui,
The 'state.backend.fs.memory-threshold' configures the threshold below
which state is stored as part of the metadata, rather than in separate
files. So as a result the JM will use its memory to merge small
checkpoint files and write them into one file. Currently the
FLIP-306[1][2] is propo
We found that for some tasks, the JM memory continued to increase. I set
the parameter of state.backend.fs.memory-threshold to 0, and the JM memory
would no longer increase, but many small files might be written in this
way. Does the community have any optimization plan for this area?
