That sounds like a good option. Might be worth stating that explicitly in the FLIP.
No other questions from me - will be a nice extension! Ryan van Huuksloot Staff Engineer, Infrastructure | Streaming Platform [image: Shopify] <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> On Thu, Aug 14, 2025 at 12:22 PM Yuepeng Pan <panyuep...@apache.org> wrote: > Hi, Hi, Ryan van Huuksloot. > > >Are you planning on having a thread to check for TTL? Or what is the plan > >for TTL? > >The quantity based would have a check when a new job is archived? > > Just like the implementation in the POC[1], if we continue following the > process where > HistoryServer#start method periodically invokes > HistoryServerArchiveFetcher#fetchArchives > based on 'historyserver.archive.fs.refresh-interval' to check > whether target files should be retained, what do you think about it ? > Of course, I'm very open to hearing about other potentially better > implementation approaches. > Please let me know what's your opinion. > Thank you. > > [1] https://github.com/apache/flink/pull/26902 > > Best, > Yuepeng Pan > > > On 2025/08/14 16:07:10 Ryan van Huuksloot wrote: > > Thanks, sounds good. > > > > Are you planning on having a thread to check for TTL? Or what is the plan > > for TTL? > > The quantity based would have a check when a new job is archived? > > > > Ryan van Huuksloot > > Staff Engineer, Infrastructure | Streaming Platform > > [image: Shopify] > > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email > > > > > > > > On Thu, Aug 14, 2025 at 12:04 PM Yuepeng Pan <panyuep...@apache.org> > wrote: > > > > > Hi, Ryan van Huuksloot. > > > > > > Thank you very much for your reply. > Question: Is the History Server > then > > > going to delete the files stored? > (i.e. we use GCS, would it delete > the > > > files there as well?) > Or is this strictly what is shown in the UI? > > > > > > > > > > > > > > > Yes, this feature introduced in the FLIP is a super-set of the original > > > feature that is controlled by 'historyserver.archive.retained-jobs'. > > > > > > So if I understand correctly, after the new feature is introduced, it > > > would affect the retention period of remote distributed storage jobs > > > history files as well, not only for what is shown in the UI. > > > > > > > > > > > > > > > Best, > > > Yuepeng Pan > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2025-08-14 23:34:54, "Ryan van Huuksloot" > > > <ryan.vanhuuksl...@shopify.com.INVALID> wrote: > > > >I took a look. Overall it would be nice to have more ways to > configure the > > > >History Server. > > > > > > > >Question: Is the History Server then going to delete the files stored? > > > >(i.e. we use GCS, would it delete the files there as well?) > > > >Or is this strictly what is shown in the UI? > > > > > > > >Ryan van Huuksloot > > > >Staff Engineer, Infrastructure | Streaming Platform > > > >[image: Shopify] > > > >< > https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > > > > > > > > > >On Thu, Aug 14, 2025 at 11:17 AM Yuepeng Pan <panyuep...@apache.org> > > > wrote: > > > > > > > >> Bumping this thread. Thanks! > > > >> > > > >> Best, > > > >> Yuepeng Pan > > > >> > > > >> On 2025/08/11 03:49:27 Yuepeng Pan wrote: > > > >> > Hi community, > > > >> > > > > >> > > > > >> > Currently, HistoryServer supports only a quantity-based job > archive > > > >> retention policy [1]. > > > >> > This is insufficient for scenarios such as: > > > >> > - Time-based retention (e.g., last X days). > > > >> > - Combined rules (e.g., within 7 days AND ≤100 jobs). > > > >> > > > > >> > > > > >> > To address these limitations, I’d like to start a discussion on > > > FLIP-490 > > > >> [2], > > > >> > which proposes a more flexible job archive retention mechanism > that > > > >> supports time-based, quantity-based, and composite strategies (with > > > AND/OR > > > >> logic). > > > >> > > > > >> > > > > >> > Looking forward to your feedback. > > > >> > > > > >> > > > > >> > Best, > > > >> > Yuepeng Pan > > > >> > > > > >> > > > > >> > [1] > > > >> > > > > https://github.com/apache/flink/blob/cae5fb4d3b6d9e0c10c3539ea4994fc1ad463b70/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServer.java#L241 > > > >> > [2] > > > >> > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=332499857 > > > >> > > > > > >
