Hello, I'm trying to build a system that will generate "last X hours" aggregates over an incoming kafka streaming.
For example, "how many events happened per user for the last 24 hours" So let's say I consume and event for User A which had 5 events in the last 24h so over, I want to emit (A,6) as soon as possible. But also, if an event goes out of the 24 hour period (as time goes by), I want to emit a lower event count for user A. I tried different approaches: 1. Tumbling windows - no good, I will only update once a day 2. Sliding windows - "can" work but in order to send updates quickly (let's say within a minute delay) I will have to have a gap of only 1 minute between windows, resulting with 1440 concurrent windows for each event, which is not recommended 3. Session windows - this is pretty close, I can make the gap 24 hours, however I need to add 2 things that Session doesn't do by default - trigger an output on each element and set and evictor to remove events that are out of the moving window So couple of questions: 1. Am I missing something and there's a better way to do it? 2. For the Session windows idea, I would need an evictor and custom trigger, I didn't see how to do it with TableAPI, I would prefer to use it so I can use FlinkSQL expressions for the actual aggregation, am I missing how to add evictors for TableAPI? Thanks ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
