Hi all, Thanks everyone for the valuable feedback and discussions on this FLIP.
Based on the discussion so far, the proposal has received generally positive feedback, and several important points have been clarified, including: - ArchiveStorage API design considerations - RocksDB deployment model and isolation between HistoryServer instances - Cleanup and retention strategy compatibility with existing mechanisms Besides, the earlier related discussion can be found here: https://lists.apache.org/thread/6thlq9c5twyvzmcw7q24nm4q0rcbz5qp If there are no further major concerns, I’m planning to start the VOTE thread next Tuesday. Please feel free to share any additional feedback before then. Best regards, Zihao zihao chen <[email protected]> 于2026年5月19日周二 21:05写道： > Hi Zuo, > > Thanks for your feedback and for aligning in this direction. > > Here are the clarifications regarding your questions: > > - *RocksDB Deployment*: > > RocksDB instance is coupled with the HistoryServer instance (each > instance has its own independent local RocksDB). There is no shared > access between multiple HistoryServer instances. > > > - *Cleanup Strategy*: > > The core cleanup still relies on the original ArchiveRetainedStrategy (max > job counts, TTL, etc.). While we've also implemented a disk-capacity-based > cleanup strategy in our internal practice to prevent disk exhaustion, > this feature is relatively independent. I decouple it for now and discuss > it > further in a follow-up FLIP. > > > Let me know if this looks good to you! > > > Best regards, > > Zihao > > > 魏祚 <[email protected]> 于2026年5月19日周二 17:33写道： > >> >> >> Hi Zihao, >> >> >> Thanks for your proposal. The excessive small files problem of >> HistoryServer is indeed a real pain point in large-scale production >> environments, and introducing RocksDB is a great idea. >> There's a few details I'd like to clarify: >> What is the deployment strategy for RocksDB? Is there a scenario where >> multiple HistoryServer instances share and access the same RocksDB >> instance? If so, are there any potential compatibility or concurrency risks? >> After introducing RocksDB, what is the strategy for cleaning up >> historical garbage files and expired job archives? >> >> >> Best regards, >> Zuo Wei >> >> >> ----- Original Message ----- >> From: "zihao chen" <[email protected]> >> To: [email protected] >> Sent: Sat, 9 May 2026 11:37:08 +0800 >> Subject: [DISCUSS] FLIP-XXX: Support Pluggable Storage Backend for >> HistoryServer >> >> Hi all, >> >> I’d like to start a discussion on FLIP-XXX: >> >> *Support Pluggable Storage Backend forHistoryServer*. >> >> This FLIP proposes improving the HistoryServer >> to address excessive *small files* when handling >> large numbers of archived jobs. >> >> [Proposal] >> Optional *RocksDB-based storage* to reduce >> small files >> >> [Compatibility] >> Full backward compatibility (FILE as default) >> >> The detailed design is described in the >> FLIP document: >> >> >> https://docs.google.com/document/d/1idHu5bq0GOsUuUAEIJSJ2UuekcDjbW0tHLNbsQfugDg/edit?usp=sharing >> >> This FLIP is split from the earlier discussion [1]. >> >> Looking forward to your feedback. >> >> [1] https://lists.apache.org/thread/6thlq9c5twyvzmcw7q24nm4q0rcbz5qp >> >> >> Best regards, >> >> Zihao Chen >> >
