RE: Notify on 0 events in a Tumbling Event Time Window

2022-05-10 Thread Schwalbe Matthias
Otto Cc: user@flink.apache.org Subject: Re: Notify on 0 events in a Tumbling Event Time Window ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Thanks Andrew. We did consider this solution too. Unfortunately we do not have permissions to generate artificial kafka events in our ecosystem

Re: Notify on 0 events in a Tumbling Event Time Window

2022-05-09 Thread Shilpa Shankar
Thanks Andrew. We did consider this solution too. Unfortunately we do not have permissions to generate artificial kafka events in our ecosystem. Dario, Thanks for your inputs. We will give your design a try. Due the number of events being processed per window, we are using incremental aggregate fu

Re: Notify on 0 events in a Tumbling Event Time Window

2022-05-09 Thread Dario Heinisch
It depends on the user case,  in Shilpa's use case it is about users so the user ids are probably know beforehand. https://dpaste.org/cRe3G <= This is an example with out an window but essentially Shilpa you would be reregistering the timers every time they fire. You would also have to ingest

Re: Notify on 0 events in a Tumbling Event Time Window

2022-05-09 Thread Andrew Otto
This sounds similar to a non streaming problem we had at WMF. We ingest all event data from Kafka into HDFS/Hive and partition the Hive tables in hourly directories. If there are no events in a Kafka topic for a given hour, we have no way of knowing if the hour has been ingested successfully. Fo

Notify on 0 events in a Tumbling Event Time Window

2022-05-09 Thread Shilpa Shankar
Hello, We are building a flink use case where we are consuming from a kafka topic and performing aggregations and generating alerts based on average, max, min thresholds. We also need to notify the users when there are 0 events in a Tumbling Event Time Windows. We are having trouble coming up with