First I was told that my application need only perform keyed aggregation of streaming IoT data on a sliding window. Flink seemed the obvious choice.
Then I was told that the window size must be configurable, taking on one of 5 possible values, anywhere from 5-60 minutes. Oh and configuration changes should take effect immediately. No biggie - I just opted to perform aggregation against all 5 possible window durations and let the post-processor worry about which outputs were of interest. Now the latest requirement (and the interesting part): if the IoT devices lose connectivity, they will buffer many days worth of data until connectivity is restored at which point all of that buffered data will be transmitted to my application. I believe this implies that event time (as determined by each individual device) must now be taken into consideration but... Question 1: is Flink really the right choice for this application now? Assuming the memory requirements for allowing such late data wouldn't be a deal-breaker, is Flink even capable of tracking event time on a per device/key basis? Question 2: Assuming a solution with Flink is suitable, what constructs would I need to leverage? Custom windows maybe? Custom triggers and evictors?
