Hi Joe, Yes, that is correct! Only when a new record arrives and we know that timestamp, we can deduce the watermark and advance it. The window operator would close the old window and open a new one.
Sorry that I haven't seen that immediately. It's sometimes hard to think in terms of individual records when you are used to think in millions. On Wed, Jun 9, 2021 at 8:49 PM Joseph Lorenzini <jlorenz...@gohealth.com> wrote: > Hi Arvid, > > > > I may have figured out the problem. > > > > When using a tumbling window on a keyed stream and event time is being > used, does time only move forward when there’s an event with a newer > timestamp? Said another way: if I have a 5 second tumbling window, the job > would need to consume at least two events before a computation would occur: > the first event has a timestamp that fits within the 5 second window, the > second event has timestamp that exceeds the max timestamp of the previous > window. > > > > Does that sound right? > > > > Thanks, > > Joe > > > > *From: *Arvid Heise <ar...@apache.org> > *Date: *Wednesday, June 9, 2021 at 8:34 AM > *To: *Joseph Lorenzini <jlorenz...@gohealth.com> > *Cc: *"user@flink.apache.org" <user@flink.apache.org> > *Subject: *Re: Records Are Never Emitted in a Tumbling Event Window When > Each Key Only Has One Record > > > > Hi Joe, > > > > could you please check (in web UI) if the watermark is advancing past the > join? The window operator would not trigger if it doesn't advance. > > On which Flink version are you running? > > > > On Tue, Jun 8, 2021 at 10:13 PM Joseph Lorenzini <jlorenz...@gohealth.com> > wrote: > > Hi all, > > > > I have observed behavior joining two keyed streams together, where events > are never emitted. The source of each stream is a different kafka topic. I > am curious to know if this expected and if there’s a way to work around it. > > > > I am using a tumbling event window. All records across the two kafka > topics occurred within the same 5 second window of time. Each kafka topic > has a single partition. For each kafka topic, I configured the flink kafka > consumer like so: > > > > consumer.assignTimestampsAndWatermarks( > > WatermarkStrategy > > > .<CustomersUnion>forBoundedOutOfOrderness(Duration.ofSeconds(10)) > > .withIdleness(Duration.ofSeconds(10)) > > ); > > > > The tumbling window has a duration of 60 seconds. Now it happens to be the > case that there is only a single event when joining on a key. If I use > Tumbling Process window then events are emitted as expected. If I actually > ensure there are multiple events for a key then the events are also > emitted. However, if it’s a single event per key in a tumbling event window > then no events are emitted. > > > > Is this expected and if it how do you handle this use case? > > > > Thanks, > > Joe > > Privileged/Confidential Information may be contained in this message. If > you are not the addressee indicated in this message (or responsible for > delivery of the message to such person), you may not copy or deliver this > message to anyone. In such case, you should destroy this message and kindly > notify the sender by reply email. Please advise immediately if you or your > employer does not consent to Internet email for messages of this kind. > Opinions, conclusions and other information in this message that do not > relate to the official business of my firm shall be understood as neither > given nor endorsed by it. > > Privileged/Confidential Information may be contained in this message. If > you are not the addressee indicated in this message (or responsible for > delivery of the message to such person), you may not copy or deliver this > message to anyone. In such case, you should destroy this message and kindly > notify the sender by reply email. Please advise immediately if you or your > employer does not consent to Internet email for messages of this kind. > Opinions, conclusions and other information in this message that do not > relate to the official business of my firm shall be understood as neither > given nor endorsed by it. >
