Hello, I’m trying to implement session windows over a set of connected streams (event time), with some custom triggering behavior. Essentially, I allow very long session gaps, but I have an “end session” event that I want to cause the window to fire and purge. I’m assigning timestamps and watermarks using BoundedOutOfOrdernessTimestampExtractor, with a 1 minute delay for the watermark. I have things mostly wired up, but I have some confusion about how I can ensure that my streams stay “in sync” relative to time.
Let’s say I am connecting streams A and B. Stream A is where the “end session” event always comes from. If I have a session involving events from time t to t’ in stream A, and then at t’ I get an “end session”, I want to ensure that the window doesn’t fire until stream B has also processed events (added events to the window) up to time t’. My understanding is that this is what the trailing watermark is for, and that in connected streams, the lowest (earliest) watermark of the input streams is what is seen as the watermark downstream. Currently, I’m setting a timer for the current time + 1 when I see my “end event”, with the idea that that timer will fire when the WATERMARK passes that time, i.e., all streams have progressed at least as far as that end event. However, the implementation of EventTimeTrigger doesn’t really look like that’s what’s going on. Can anyone clear up how these concepts interact? Is there a good model for this “session end event” concept that I can take a look at? Thanks, Andrew -- *Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
