Hi Matt, Generally speaking, using state with TTL in Flink should not differ a lot from just using Flink with state [1]. You have to provision your system so that it can keep the state of size which is worth of 7 days.
The existing Flink state backends provide background cleanup to automatically remove the expired state eventually, so that your application does not need to do any explicit access of the expired state to clean it. The background cleanup is active by default since Flink 1.10 [2]. Enabling TTL for state, of course, comes for price because you need to store timestamp and spend CPU cycles for the background cleanup. This affects storage size and potentially processing latency per record. You can read about details and caveats in the docs: for heap state [3] and RocksDB [4]. Best, Andrey [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/state.html#state-time-to-live-ttl [2] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/state.html#cleanup-of-expired-state [3] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/state.html#incremental-cleanup [4] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/state.html#cleanup-during-rocksdb-compaction On Thu, Mar 19, 2020 at 6:48 PM Matt Magsombol <raffy4...@gmail.com> wrote: > Suppose I'm using state stored in-memory that has a TTL of 7 days max. > Should I run into any issues with state this long other than potential OOM? > > Let's suppose I extend this such that we add rocksdb...any concerns with > this with respect to maintenance? > > Most of the examples that I've been seeing seem to pair state with > timewindows but I'll only read from this state every 15 seconds ( or some > small timewindow ). After each timewindow, I *won't* be cleaning up the > data within the state b/c I'll need to re-lookup from this state on future > time windows. I'll effectively rely on TTL based on key expiration time and > I was wondering what potential issues I should watch out for this. >
