Second approach is good to try out. I am also solving for a similar problem using this approach only.
Thanks Sachin On Mon, 12 May 2025 at 3:52 PM, Ehud Lev <ehud....@forter.com> wrote: > Hi Flink community, > > We're building a Flink topology that aggregates events by key. When a key > is seen for the first time, we load its base state from an external store. > After performing some calculations, we emit the result to Kafka, and > another process is responsible for writing it back to the store. > > Our goals are: > > 1. > > *Keep per-key state alive for a few hours*, but *remove it if no new > events arrive during that time*. > 2. > > *Suppress frequent updates* and *emit output only every 5 seconds*, > but *only if updates occurred* during that interval. > > I see two possible implementation strategies: > > 1. > > *Session Windows (processing time):* > This would allow automatic state cleanup based on inactivity, but I'm > concerned about *merging session states*, since our logic depends on > having a well-defined "base state" and applying updates to it. I can not > handle merge basically. > 2. > > *KeyedProcessFunction with timers:* > In this approach, we’d maintain keyed state, and use: > - > > A *5-second periodic timer* to check for and emit updates (if any). > - > > That same timer can also track *last-seen timestamps* to expire > state after a few idle hours. > > *My questions:* > Is the session window approach even suitable here, given the merging issue > I have? > Is the KeyedProcessFunction with a "single recurring timer" a recommended > pattern in such use cases? > Are there any caveats or better alternatives for managing "suppressed + > expiring" keyed state? > > > Thanks in advance! > > -- > Ehud Lev, Staff Engineer > email: ehud....@forter.com web: www.forter.com > mobile: 052-5832253 >
