Hi, I have an upstream operator that outputs device state transition messages with event timestamps. Meaning it only emits output when a transition takes place. For example, state1 @ 1 PM state2 @ 2 PM and so on.
*Using a downstream operator, I want to emit notification messages as per some configured periodicity.* For example, if periodicity = 20 min, in the above scenario this operator will output : state1 notification @ 1PM state1 notification @ 1.20PM state1 notification @ 1.40PM ... *Now the main issue is that I want this to be driven by the watermark and not by transition events received from upstream. *Meaning I would like to see notification events as soon as the watermark crosses their timestamps; *not* when the next transition event arrives at the operator (which could be hours later, as above). My first solution, using a keyedProcessFunction and timers did not work as expected because the order in which transition events arrived at this operator was non-deterministic. To elaborate, assume a setAutoWatermarkInterval of 10 second. If we get transition events : state1 @ 1sec state2 @ 3 sec state3 @ 5 sec state1 @ 8 sec the order in which these events arrived at my keyedProcessFunction was not fixed. To solve this, these messages need to be sorted on event time, which led me to my second solution. My second solution, using a EventTimeTumblingWindow with size = setAutoWatermarkInterval, also does not work. I sorted accumulated events in the window and applied notification-generation logic on them in order. However, I assumed that windows are created even if there are no elements. Since this is not the case, this solution generates notifications only when the next state tranisition message arrives, which could be hours later. Does anyone have any suggestions on how I can implement this? Thanks!
