So my TTL config is like:
StateTtlConfig.newBuilder(Duration.ofHours(1))
.setUpdateType(StateTtlConfig.UpdateType.OnReadAndWrite)
.setStateVisibility(StateTtlConfig.StateVisibility.ReturnExpiredIfNotCleanedUp)
.build();
Issue is that if every time I use ListState.update it would be an expensive
process to update the entire list of hashes everytime.
I can use Map State but how can I control the size of this map state to not
grow beyond a certain point ?
Use case is that for a given key we get data every few seconds. Each data
has a unique hash associated with it.
We want to check if a certain hash is in the state, it should not process
that data again.
At the same time old hashes expire beyond a certain ttl, so that the hashes
state don't grow beyond a size.
I believe even with map state I have to expire the entire map and there is
no way to set ttl for individual keys in the map.
Please let me know if there is a better way to do this.
Thanks
Sachin
On Fri, May 30, 2025 at 9:32 AM Zakelly Lan <zakelly....@gmail.com> wrote:
> Hi Sachin,
>
> I assume you are using the rocksdb state backend. The TTL for ListState is
> applied for each list entry, if you are using `ListState.add`. However if
> you do ListState.update, the entire list is rewrite so the ttl is updated.
> Could you share your use case and the ttl config?
> Another suggestion is to use the Map State, thus you could manipulate each
> entry freely without rewriting the entire list.
>
>
> Best,
> Zakelly
>
> On Fri, May 30, 2025 at 12:22 AM Sachin Mittal <sjmit...@gmail.com> wrote:
>
>> Hi,
>> I think ttl would be applied for the entire list,
>> I would like the ListState to restrict the entries by size and
>> automatically purge older added entries as new ones get added.
>> Something similar to a bounded list.
>>
>> Thanks
>> Sachin
>>
>>
>> On Thu, May 29, 2025 at 6:51 PM Sigalit Eliazov <e.siga...@gmail.com>
>> wrote:
>>
>>> hi,
>>> i think you can achieve this by using StateTtlConfig to define the
>>> ttl, and add ListStateDescriptor to the ListState definition.
>>>
>>> thanks,
>>> Sigalit
>>>
>>>
>>>
>>> On Thu, May 29, 2025 at 11:53 AM Sachin Mittal <sjmit...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I am adding some hashes of my elements in my list date to check for
>>>> dedups.
>>>>
>>>> Now to not have an infinte growing list, I would like to limit the size
>>>> of hashes in that list to say a number or just add some TTL config which
>>>> would expire the entries in the list beyond certain time.
>>>>
>>>> Is this something possible using Flink constructs.
>>>>
>>>> If not, is there any way I can achieve this ?
>>>>
>>>> Thanks
>>>> Sachin
>>>>
>>>>
