[
https://issues.apache.org/jira/browse/FLINK-10473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725757#comment-16725757
]
Andrey Zagrebin commented on FLINK-10473:
-----------------------------------------
Hi [~karmagyz], I have couple of questions to your concerns and maybe answers:
* Do you mean by LRU strategy that later written state entries will be cleaned
later? true, *keyNamespaceIterator* iterates state entries in the order of
their distribution by key, basically randomly from the time perspective, but it
checks expiration and cleans up entries according to the timestamp of the last
update. The TTL feature provides guarantees of never cleaning unexpired records
and cleaning expired records eventually based on the best effort. To cleanup
entries more aggressively, timers are better solution but they come for storage
price because they require sorting by time and we have only by key in backends.
From what I've heard lazy cleanup could be sufficient for SQL use cases but I
do not know this component. [~fhue...@gmail.com], [~tiwalter], [~pnowojski],
what do you think?
* This effort is related to TTL feature and time-based eviction of state
entries. I agree, that *MaximumSize* can be a useful feature but this sounds
like another effort.
* As *cleanupSize* is usually greater than 1 record, it means that for every
state update operation (which potentially creates new keys and increases size)
cleanup will check several entries. it should guarantee that cleanup is faster
than update rate and state size should not grow. It comes for price of
increasing record processing latency. `Latency price vs cleanup rate' is
adjusted by *cleanupSize*. I have end to end test on one of my branches which
shows how state size stabilises over time with certain update rate but no
explicit deletion by application, only TTL cleanup
(https://github.com/azagrebin/flink/blob/ttl_incremental_cleanup/flink-end-to-end-tests/flink-stream-state-ttl-inc-cleanup-test/src/main/java/org.apache.flink.streaming.tests/DataStreamStateTtlIncCleanupTestProgram.java).
'
> State TTL incremental cleanup using Heap backend key iterator
> -------------------------------------------------------------
>
> Key: FLINK-10473
> URL: https://issues.apache.org/jira/browse/FLINK-10473
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
> Affects Versions: 1.7.0
> Reporter: Andrey Zagrebin
> Assignee: Andrey Zagrebin
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> This feature enables lazy background cleanup of state with time-to-live in
> state keyed backend which stores state in JVM heap. The idea is to keep a
> global state lazy iterator with loose consistency. Every time a state value
> for some key is accessed or a record is processed, the iterator is advanced,
> TTL of iterated state entries is checked and the expired entries are cleaned
> up. When the iterator reaches the end of state storage it just starts over.
> This way the state with TTL is regularly cleaned up to prevent ever growing
> memory consumption. The caveat of this cleanup strategy is that if state is
> not accessed or no records are processed then accumulated expired state still
> occupies the storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
