Hi Ammon, The duration is per item, and the cleanup happens transparently and incrementally via RocksDB (background compactions with a custom filter) [1]
In your example a gets cleaned up, while b will be cleaned in ~10min. Kind regards, Igal. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/state/#cleanup-during-rocksdb-compaction On Wed, Jul 14, 2021 at 1:59 PM Igal Shilman <i...@ververica.com> wrote: > Hi Ammon, > > The duration is per item, and the cleanup happens transparently and > incrementally via RocksDB (background compactions with a custom filter) [1] > > In your example a gets cleaned up, while b will be cleaned in ~10min. > > Kind regards, > Igal. > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/state/#cleanup-during-rocksdb-compaction > > > On Wed, Jul 14, 2021 at 1:53 PM Igal Shilman <i...@ververica.com> wrote: > >> Hi Ammon, >> >> The duration is per item, and the cleanup happens transparently and >> incrementally via RocksDB (background compactions with a custom filter) [1] >> >> In your example a gets cleaned up, while b will be cleaned in ~10min. >> >> Kind regards, >> Igal. >> >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/state/#cleanup-during-rocksdb-compaction >> >> On Wed, Jul 14, 2021 at 5:09 AM Ammon Diether <adiet...@gmail.com> wrote: >> >>> >>> Thank you for asking. I meant PersistedTable >>> -> >>> https://github.com/apache/flink-statefun/blob/release-3.0/statefun-sdk-embedded/src/main/java/org/apache/flink/statefun/sdk/state/PersistedTable.java >>> >>> It is related to state backend. I am using rocksdb backend. >>> >>> >>> >>> On Tue, Jul 13, 2021 at 8:53 PM Caizhi Weng <tsreape...@gmail.com> >>> wrote: >>> >>>> Hi >>>> >>>> By PersistentTable do you mean state backend? If yes, the answer >>>> differs with different operators and state backends. >>>> >>>> For keyed states the duration is for per key. However the exact time to >>>> clean up a key really depends on the operator and the state backend. Most >>>> operators will register a timer and when the timer triggers it cleans up >>>> that key. >>>> >>>> Ammon Diether <adiet...@gmail.com> 于2021年7月14日周三 上午5:17写道： >>>> >>>>> >>>>> Question >>>>> If the duration is 20 minutes, >>>>> 1) is the duration per item? >>>>> 2) or is the duration for the table as a whole? >>>>> >>>>> Suppose the following items >>>>> ("a", "a-value") 30 minutes ago >>>>> ("b", "b-value") 10 minutes ago >>>>> >>>>> Does "a" get cleaned up? or neither gets cleaned up yet because the >>>>> most recent item is not above the duration? >>>>> >>>>> Another question >>>>> Also, if enough time passes, does the cleanup happen during a >>>>> checkpoint, savepoint, or the next time the PersistentTable is accessed? >>>>> If the table is never accessed again, will it remain in the savepoints >>>>> forever? >>>>> >>>>>
