Flink MapState manages the TTL of each map entry independently. If you add/update a “new hash”, the “old hash” that was written one hour ago stays unaffected and can still expire as it should be. So it should meet your requirements of deduplication and size-control.
> 2025年6月4日 10:54，Sachin Mittal <sjmit...@gmail.com> 写道： > > So my TTL config is like: > StateTtlConfig.newBuilder(Duration.ofHours(1)) > .setUpdateType(StateTtlConfig.UpdateType.OnReadAndWrite) > > .setStateVisibility(StateTtlConfig.StateVisibility.ReturnExpiredIfNotCleanedUp) > .build(); > > Issue is that if every time I use ListState.update it would be an expensive > process to update the entire list of hashes everytime. > > I can use Map State but how can I control the size of this map state to not > grow beyond a certain point ? > > Use case is that for a given key we get data every few seconds. Each data has > a unique hash associated with it. > > We want to check if a certain hash is in the state, it should not process > that data again. > At the same time old hashes expire beyond a certain ttl, so that the hashes > state don't grow beyond a size. > > I believe even with map state I have to expire the entire map and there is no > way to set ttl for individual keys in the map. > > Please let me know if there is a better way to do this. > > Thanks > Sachin > > > > > > On Fri, May 30, 2025 at 9:32 AM Zakelly Lan <zakelly....@gmail.com > <mailto:zakelly....@gmail.com>> wrote: > Hi Sachin, > > I assume you are using the rocksdb state backend. The TTL for ListState is > applied for each list entry, if you are using `ListState.add`. However if you > do ListState.update, the entire list is rewrite so the ttl is updated. Could > you share your use case and the ttl config? > Another suggestion is to use the Map State, thus you could manipulate each > entry freely without rewriting the entire list. > > > Best, > Zakelly > > On Fri, May 30, 2025 at 12:22 AM Sachin Mittal <sjmit...@gmail.com > <mailto:sjmit...@gmail.com>> wrote: > Hi, > I think ttl would be applied for the entire list, > I would like the ListState to restrict the entries by size and automatically > purge older added entries as new ones get added. > Something similar to a bounded list. > > Thanks > Sachin > > > On Thu, May 29, 2025 at 6:51 PM Sigalit Eliazov <e.siga...@gmail.com > <mailto:e.siga...@gmail.com>> wrote: > hi, > i think you can achieve this by using StateTtlConfig to define the ttl, and > add ListStateDescriptor to the ListState definition. > > thanks, > Sigalit > > > > On Thu, May 29, 2025 at 11:53 AM Sachin Mittal <sjmit...@gmail.com > <mailto:sjmit...@gmail.com>> wrote: > Hi, > I am adding some hashes of my elements in my list date to check for dedups. > > Now to not have an infinte growing list, I would like to limit the size of > hashes in that list to say a number or just add some TTL config which would > expire the entries in the list beyond certain time. > > Is this something possible using Flink constructs. > > If not, is there any way I can achieve this ? > > Thanks > Sachin >
