Hi folks, Just a gentle reminder regarding the FLIP I proposed for improving the Flink History Server. Thanks for your time and attention.
Best, - Allison On Thu, Jan 16, 2025 at 4:06 PM Allison <achang5...@gmail.com> wrote: > Hi everyone, > > I would like to initiate a discussion for the FLIP below, which enhances > to the Flink History Server to allow greater scalability of the service. > > Motivation: > > Currently, the Flink History Server (FHS) is limited in the number of job > archives it can serve based on the storage capacity of the node that the > FHS runs in. Job archives are stored locally in a cache which creates a > local directory which is expanded out based on the contents of a single > json archive file. This not only uses up local memory space, but also > because of how the FHS expands the job archives into a nested directory > structure, for jobs with a large number of taskmanagers or subtasks, inode > space often runs out. In order to make the FHS more performant, we would > like to introduce the ability to decouple the job archive storage for the > FHS from being limited to the local cache, to being able to store and fetch > jobs archives from a remote file store. > > FLIP proposal document: > https://cwiki.apache.org/confluence/display/FLINK/FLIP+505%3A+Flink+History+Server+Scability+Improvements%2C+Remote+Data+Store+Fetch+and+Per+Job+Fetch > > Thanks! > > Best, > - Allison Chang >
