Hi Becket, Thanks for the feedback! I’ve updated the design doc with a new “Granularity of the Early Fire Hint” section: link <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.gle8q83bjlj5> .
Here’s a quick summary: - EARLY_FIRE is scoped to a single operator. If multiple operators in one query can support early firing, Flink applies the hint to the first matching operator and logs a warning to highlight the ambiguity. Subsequent operators will not receive the hint. - Split queries for different early-fire settings. If you need separate early-fire configurations - or just want to avoid ambiguity - splitting the SQL into multiple statements or views ensures each operator has its own distinct EARLY_FIRE hint. Please let me know if you have any other questions or suggestions. Best, Weiqing On Fri, Feb 28, 2025 at 5:01 PM Becket Qin <becket....@gmail.com> wrote: > Thanks for updating the FLIP, Weiqing. > > Another question, what is the granularity of the early fire hint? Will the > hint be applied to all the outer joins in the same block? What if in the > query block there are also aggregations? For example: > > SELECT window_start, window_end, buyer, count(*) /*+ > EARLY_FIRE('delay'='5s' > , 'time_mode'='rowtime|proctime') */ > FROM Orders o LEFT JOIN Shipments s > WHERE o.id = s.order_id > AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time > GROUP BY o.window_start, o.window_end, buyer; > > What will be the behavior? Will both the aggregation and interval join? Or > will it only be applied to the join? > It would be useful to define the behavior here. > > Thanks, > > Jiangjie (Becket) Qin > > On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <yangweiqing...@gmail.com> > wrote: > > > Thanks for the feedback, Becket. I’ve heard similar concerns from other > > reviewers, and I now agree that the `interval` option is likely not > widely > > used. In the FLIP, it was intended specifically for Interval Join, not > as a > > general early-fire mechanism. We can consider making it more generic in a > > future update. Since it’s optional, I’ll go ahead and remove it for > > clarity. > > > > Appreciate your input! > > > > Best, > > Weiqing > > > > On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <becket....@gmail.com> wrote: > > > > > Sorry for the late reply. I have a question regarding the configuration > > of > > > interval. I am wondering in which scenario the interval config will > > > actually be used. Is the interval configuration only useful for > windowed > > > aggregation? > > > > > > For the join operations, early fire seems only applicable to the outer > > join > > > before there is a match. Once there is a match, the output of the join > > > operation should be just event driven. For example, consider the > > following > > > scenario: > > > 1. left outer join, and there is no match from the right side before > the > > > initial delay > > > 2. a record [left, null] was emitted due to early fire > > > 3. one of the following two cases must happen > > > a. there is still no match event from the right side before the > > > specific interval passes. In this case, we are not supposed to emit > > another > > > [left, null]. > > > b. if there is a match, a record of [left, right] should be emitted > > > immediately regardless of the interval. If this happens, the interval > > will > > > be ignored from this point on. > > > > > > That said, if the configuration proposed in this FLIP is intended not > > only > > > for join, but for general purpose early fire, then the interval config > > > makes sense. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <yangweiqing...@gmail.com > > > > > wrote: > > > > > > > Thank you all for reviewing. As there are no objections, I’ll move > > > forward > > > > with a vote. > > > > > > > > Best regards, > > > > Weiqing > > > > > > > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan < > > > > vsowr...@asu.edu> > > > > wrote: > > > > > > > > > Same here, I don't have any more questions. Thanks! > > > > > > > > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xingc...@gmail.com> > > wrote: > > > > > > > > > > > Hi Weiqing, > > > > > > > > > > > > I don't have any more questions. The doc looks good to me. > > > > > > > > > > > > Thanks, > > > > > > Xingcan > > > > > > > > > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan < > > > > > > vsowr...@asu.edu> > > > > > > wrote: > > > > > > > > > > > > > Hi Weiqing, > > > > > > > > > > > > > > Thanks, that makes sense! Looks like I missed it. > > > > > > > > > > > > > > Regards > > > > > > > Venkata krishnan > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang < > > > > > yangweiqing...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Venkata, > > > > > > > > > > > > > > > > * > where only one earlyFire is fired The DELAY * > > > > > > > > > > > > > > > > The DELAY option mentioned in the Public Interfaces section > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$ > > > > > > > > > > > > > > > > > of the proposal can produce a single early fire per interval, > > > > > aligning > > > > > > > with > > > > > > > > your suggestion. How it integrates with existing early-fire > > > > > > > configurations > > > > > > > > are mentioned here > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$ > > > > > > > > >. > > > > > > > > Let me know if you have any further questions or feedback! > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Weiqing > > > > > > > > > > > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan < > > > > > > > > vsowr...@asu.edu> wrote: > > > > > > > > > > > > > > > > > Thanks for the response! > > > > > > > > > > > > > > > > > > > If the optional `interval` in the proposal is enabled, > > > > early-fire > > > > > > > > outputs > > > > > > > > > will occur repeatedly at every earlyFireInterval, not just > > > once. > > > > > > After > > > > > > > > that > > > > > > > > > interval concludes, there will also be a final emission. > > > > > > > > > > > > > > > > > > One more question, would it make sense to support a > mechanism > > > > > through > > > > > > > > > configuration where only one earlyFire is fired for an > > interval > > > > or > > > > > > > > window? > > > > > > > > > This would also simplify the state overhead for use cases > > where > > > > > only > > > > > > > one > > > > > > > > > early fire is required within a window or interval. > Thoughts? > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > Venkata krishnan > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang < > > > > > > yangweiqing...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Venkata, > > > > > > > > > > > > > > > > > > > > Thanks for your feedback! > > > > > > > > > > > > > > > > > > > > If the optional `interval` in the proposal is enabled, > > > > early-fire > > > > > > > > outputs > > > > > > > > > > will occur repeatedly at every earlyFireInterval, not > just > > > > once. > > > > > > > After > > > > > > > > > that > > > > > > > > > > interval concludes, there will also be a final emission. > > > > > > > > > > > > > > > > > > > > We haven’t included a late-fire mechanism for interval > > joins > > > in > > > > > > this > > > > > > > > > FLIP, > > > > > > > > > > but it’s certainly something we can consider for future > > > > efforts! > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > Weiqing > > > > > > > > > > > > > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan > Sowrirajan > > < > > > > > > > > > > vsowr...@asu.edu> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Weiqing, > > > > > > > > > > > > > > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also > > instead > > > > of > > > > > > > > > > controlling > > > > > > > > > > > and configuring through Flink configs unlike the older > > > window > > > > > > > > > > aggregation, > > > > > > > > > > > hints seems to be a much better approach. This enables > a > > > > > > > customizable > > > > > > > > > > early > > > > > > > > > > > fire behavior for individual interval joins. > > > > > > > > > > > > > > > > > > > > > > Couple of questions: > > > > > > > > > > > > > > > > > > > > > > 1. Does the *early fire* emit an output every > > > > earlyFireInterval > > > > > > > time > > > > > > > > or > > > > > > > > > > > will it be a one time output emission and another > output > > > > > emitted > > > > > > at > > > > > > > > the > > > > > > > > > > end > > > > > > > > > > > of the interval? > > > > > > > > > > > 2. Are there plans to support *late fire *similar to > the > > > > window > > > > > > > > > > > aggregations in later FLIPs? > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > Venkata krishnan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang < > > > > > > > > yangweiqing...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks for reviewing, Xuyang! > > > > > > > > > > > > > > > > > > > > > > > > Xingcan (@xingc...@gmail.com) – do you have any > > > concerns? > > > > > > > > > > > > > > > > > > > > > > > > If no further objections arise from anyone, I’ll > > proceed > > > to > > > > > > mark > > > > > > > > FLIP > > > > > > > > > > as > > > > > > > > > > > > ready for voting. > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Weiqing > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang < > > > xyzhong...@163.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > LGTM overall. Thanks for updating. I have no > problem > > > and > > > > +1 > > > > > > for > > > > > > > > > this > > > > > > > > > > > > > feature. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > Best! > > > > > > > > > > > > > Xuyang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 在 2025-01-15 12:33:16,"Weiqing Yang" < > > > > > > yangweiqing...@gmail.com > > > > > > > > > > > > > > > > > 写道: > > > > > > > > > > > > > >Hi Xuyang, > > > > > > > > > > > > > > > > > > > > > > > > > > > >Thank you for your detailed feedback! I’ve updated > > the > > > > > > > proposal > > > > > > > > > doc > > > > > > > > > > > > > >accordingly. Please feel free to take another look > > and > > > > let > > > > > > me > > > > > > > > know > > > > > > > > > > if > > > > > > > > > > > > you > > > > > > > > > > > > > >have any further thoughts or suggestions. > > > > > > > > > > > > > > > > > > > > > > > > > > > >Best regards, > > > > > > > > > > > > > >Weiqing > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang < > > > > > xyzhong...@163.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi, Weiqing. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> After reading the new FLIP, I have no issues > with > > > the > > > > > part > > > > > > > > > `public > > > > > > > > > > > > > >> interface`. I only have some questions regarding > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> the details in the Proposed Changes section. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the > > > > > > IntervalJoin > > > > > > > > > node, > > > > > > > > > > > > IIUC: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind > of > > > the > > > > > > > > > IntervalJoin > > > > > > > > > > > can > > > > > > > > > > > > > be > > > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely > on > > > the > > > > > > > > > > requirements > > > > > > > > > > > of > > > > > > > > > > > > > the > > > > > > > > > > > > > >> sink. And the ModifyKind is always ALL. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> - When early firing is disabled, the UpdateKind > of > > > the > > > > > > > > > > IntervalJoin > > > > > > > > > > > is > > > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> - Nevertheless, whether early firing is enabled > or > > > > > > disabled, > > > > > > > > the > > > > > > > > > > > > > >> IntervalJoin should always require its input to > > keep > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with > > > NONE. > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> -- > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Best! > > > > > > > > > > > > > >> Xuyang > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" < > > > > > > > > > yangweiqing...@gmail.com> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> >Hi Xingcan and Xuyang, > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >Thanks so much for the feedback - it was very > > > > helpful! > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >*> 1. The current output stream of a time > > interval > > > > > outer > > > > > > > join > > > > > > > > > is > > > > > > > > > > an > > > > > > > > > > > > > >> >append-only stream. This change will make it a > > > > > potential > > > > > > > > > > > retractable > > > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a > > > > dynamic > > > > > > > output > > > > > > > > > > type > > > > > > > > > > > > like > > > > > > > > > > > > > >> >that. Could you add this part to your design > > doc?* > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > - Yes, enabling early firing on time > interval > > > > outer > > > > > > > joins > > > > > > > > > can > > > > > > > > > > > emit > > > > > > > > > > > > > >> > retractions when previously emitted rows are > > > > updated > > > > > > or > > > > > > > > > > > > invalidated > > > > > > > > > > > > > by > > > > > > > > > > > > > >> > later matches. I’ve updated the proposal > > > (Planner > > > > > > > > Awareness > > > > > > > > > > > > > >> > < > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$ > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > and Changes in > > > FlinkChangelogModeInferenceProgram > > > > > > > > > > > > > >> > < > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$ > > > > > > > > > > > > > >> >) > > > > > > > > > > > > > >> > to clarify that the stream might switch from > > > > > > append-only > > > > > > > > to > > > > > > > > > a > > > > > > > > > > > > > >> > retract/upsert stream. Let me know if > anything > > > is > > > > > > > missing. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >*> 2. What's the use case when the downstream > > > > > components > > > > > > > need > > > > > > > > > to > > > > > > > > > > > get > > > > > > > > > > > > > the > > > > > > > > > > > > > >> >early fired results regularly?* > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > - The new INTERVAL option (in addition to > > DELAY) > > > > > > allows > > > > > > > > > > periodic > > > > > > > > > > > > > >> updates > > > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial > > > delay. > > > > > This > > > > > > > > > > captures > > > > > > > > > > > > how > > > > > > > > > > > > > >> results > > > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s > > > > > > “Repeatedly” > > > > > > > > > > option. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >*> 3. The time interval join operator itself is > > not > > > > > quite > > > > > > > > > > efficient > > > > > > > > > > > > > when > > > > > > > > > > > > > >> >the state becomes large. Will there be any > extra > > > > > overhead > > > > > > > > after > > > > > > > > > > > > > >> introducing > > > > > > > > > > > > > >> >this feature?* > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > - Early fire does introduce some overhead by > > > > > > potentially > > > > > > > > > > > emitting > > > > > > > > > > > > > >> > partial matches multiple times with > retraction > > > > > > (avoiding > > > > > > > > > > > duplicate > > > > > > > > > > > > > >> outputs > > > > > > > > > > > > > >> > though). However, if it’s disabled, there is > > no > > > > > > > additional > > > > > > > > > > cost. > > > > > > > > > > > > > Most > > > > > > > > > > > > > >> users > > > > > > > > > > > > > >> > find the performance trade-off acceptable > for > > > the > > > > > > > > real-time > > > > > > > > > > > > > insights it > > > > > > > > > > > > > >> > provides. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >*> 1. Currently, there are some configs related > > to > > > > > early > > > > > > > > firing > > > > > > > > > > > > > available > > > > > > > > > > > > > >> >to users: > `table.exec.emit.early-fire.en**abled` > > > and > > > > > > > > > > > > > >> >`table.exec.emit.early-fire.de < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$ > > > > > > > > > > > > > >> >**lay`. > > > > > > > > > > > > > >> >Although their documentation states that they > are > > > > only > > > > > > > > > applicable > > > > > > > > > > > to > > > > > > > > > > > > > the > > > > > > > > > > > > > >> >Window operator, it seems possible that they > may > > > also > > > > > be > > > > > > > > > relevant > > > > > > > > > > > in > > > > > > > > > > > > > the > > > > > > > > > > > > > >> >context of this FLIP. Otherwise, having > different > > > > early > > > > > > > > firing > > > > > > > > > > > > > behaviors > > > > > > > > > > > > > >> >for different operators could confuse users.* > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to > avoid > > > > > > > confusion. > > > > > > > > > I’ve > > > > > > > > > > > > > added a > > > > > > > > > > > > > >> > section > > > > > > > > > > > > > >> > < > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$ > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >in > > > > > > > > > > > > > >> > the proposal highlighting that we should > align > > > > > > > hint-based > > > > > > > > > > > interval > > > > > > > > > > > > > join > > > > > > > > > > > > > >> > configurations with the existing > > > table.exec.emit.* > > > > > > > > settings. > > > > > > > > > > > > > >> Suggestions > > > > > > > > > > > > > >> > on how to make the unification are welcome! > We > > > > plan > > > > > to > > > > > > > > > extend > > > > > > > > > > > > early > > > > > > > > > > > > > >> firing > > > > > > > > > > > > > >> > to window joins via hints in a future FLIP. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent. > > > Similar > > > > > to > > > > > > > > point > > > > > > > > > 1, > > > > > > > > > > > > > perhaps > > > > > > > > > > > > > >> >we can introduce it to other window-related > > > operators > > > > > in > > > > > > > the > > > > > > > > > > > future.> > > > > > > > > > > > > > 3. > > > > > > > > > > > > > >> >You need to modify the > > > > > FlinkChangelogModeInferenceProgram > > > > > > > to > > > > > > > > > > ensure > > > > > > > > > > > > > that > > > > > > > > > > > > > >> >downstream nodes of interval joins with early > > > firing > > > > > > > enabled > > > > > > > > > are > > > > > > > > > > > > aware > > > > > > > > > > > > > of > > > > > > > > > > > > > >> >retract or upsert messages.* > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > - We agree that time_mode could be > introduced > > to > > > > > other > > > > > > > > > > > > window-based > > > > > > > > > > > > > >> > operators down the road. We also want to > > support > > > > > early > > > > > > > > fire > > > > > > > > > > for > > > > > > > > > > > > > >> > window join. Also, thanks for highlighting > > > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added > > the > > > > code > > > > > > > > change > > > > > > > > > on > > > > > > > > > > > it > > > > > > > > > > > > > >> > < > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$ > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > in the proposal. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve > > > updated > > > > > the > > > > > > > > > > proposal > > > > > > > > > > > > with > > > > > > > > > > > > > >> >these points. Please let me know if there’s > > > anything > > > > > > else I > > > > > > > > > > should > > > > > > > > > > > > > >> address. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >Best, > > > > > > > > > > > > > >> >Weiqing > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang < > > > > > > xyzhong...@163.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this > FLIP. > > I > > > > > have a > > > > > > > few > > > > > > > > > > > > > questions: > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> 1. Currently, there are some configs related > to > > > > early > > > > > > > > firing > > > > > > > > > > > > > available > > > > > > > > > > > > > >> to > > > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` > and > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although > > > their > > > > > > > > > > documentation > > > > > > > > > > > > > states > > > > > > > > > > > > > >> >> that they are only applicable to the Window > > > > operator, > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> it seems possible that they may also be > > relevant > > > in > > > > > the > > > > > > > > > context > > > > > > > > > > > of > > > > > > > > > > > > > this > > > > > > > > > > > > > >> >> FLIP. Otherwise, having different early > firing > > > > > > behaviors > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> for different operators could confuse users. > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent. > > > Similar > > > > to > > > > > > > point > > > > > > > > > 1, > > > > > > > > > > > > > perhaps > > > > > > > > > > > > > >> we > > > > > > > > > > > > > >> >> can introduce it to other window-related > > > operators > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> in the future. > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> 3. You need to modify the > > > > > > > > FlinkChangelogModeInferenceProgram > > > > > > > > > to > > > > > > > > > > > > > ensure > > > > > > > > > > > > > >> >> that downstream nodes of interval joins with > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> early firing enabled are aware of retract or > > > upsert > > > > > > > > messages. > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> -- > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> Best! > > > > > > > > > > > > > >> >> Xuyang > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" < > > > > > > > xingc...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> >> >Hi Weiqing, > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early > > fire > > > > for > > > > > > time > > > > > > > > > > > interval > > > > > > > > > > > > > outer > > > > > > > > > > > > > >> >> >joins is feasible overall. I have a few > > > questions. > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >1. The current output stream of a time > > interval > > > > > outer > > > > > > > join > > > > > > > > > is > > > > > > > > > > an > > > > > > > > > > > > > >> >> >append-only stream. This change will make > it a > > > > > > potential > > > > > > > > > > > > retractable > > > > > > > > > > > > > >> >> >stream. I'm not sure if the planner > supports a > > > > > dynamic > > > > > > > > > output > > > > > > > > > > > type > > > > > > > > > > > > > like > > > > > > > > > > > > > >> >> >that. Could you add this part to your design > > > doc? > > > > > > > > > > > > > >> >> >2. What's the use case when the downstream > > > > > components > > > > > > > need > > > > > > > > > to > > > > > > > > > > > get > > > > > > > > > > > > > the > > > > > > > > > > > > > >> >> early > > > > > > > > > > > > > >> >> >fired results regularly? > > > > > > > > > > > > > >> >> >3. The time interval join operator itself is > > not > > > > > quite > > > > > > > > > > efficient > > > > > > > > > > > > > when > > > > > > > > > > > > > >> the > > > > > > > > > > > > > >> >> >state becomes large. Will there be any extra > > > > > overhead > > > > > > > > after > > > > > > > > > > > > > introducing > > > > > > > > > > > > > >> >> >this feature? > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >Thanks, > > > > > > > > > > > > > >> >> >Xingcan > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang > < > > > > > > > > > > > > > yangweiqing...@gmail.com> > > > > > > > > > > > > > >> >> >wrote: > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >> Hi all, > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the > > proposal > > > I > > > > > > shared > > > > > > > > on > > > > > > > > > > > early > > > > > > > > > > > > > fire > > > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d > > > > greatly > > > > > > > > > appreciate > > > > > > > > > > > > your > > > > > > > > > > > > > >> >> feedback > > > > > > > > > > > > > >> >> >> or suggestions. > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> Here’s the link to the proposal document: > > Link > > > > > > > > > > > > > >> >> >> < > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$ > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> Thank you! > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> Best, > > > > > > > > > > > > > >> >> >> Weiqing > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing > > Yang > > > < > > > > > > > > > > > > > >> yangweiqing...@gmail.com > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> >> >> wrote: > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> >> > Hi all, > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about > > > > > > introducing > > > > > > > > > early > > > > > > > > > > > fire > > > > > > > > > > > > > >> support > > > > > > > > > > > > > >> >> >> for > > > > > > > > > > > > > >> >> >> > Flink SQL interval joins. > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > *Motivation* > > > > > > > > > > > > > >> >> >> > In many streaming applications, > > particularly > > > > > > > real-time > > > > > > > > > > > > analytics > > > > > > > > > > > > > >> and > > > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to > > obtain > > > > > > partial > > > > > > > > > > results > > > > > > > > > > > > > >> earlier > > > > > > > > > > > > > >> >> >> rather > > > > > > > > > > > > > >> >> >> > than waiting for full join conditions to > > > > > finalize. > > > > > > > For > > > > > > > > > > Flink > > > > > > > > > > > > SQL > > > > > > > > > > > > > >> >> interval > > > > > > > > > > > > > >> >> >> > joins, results are typically delayed > until > > > > > > > watermarks > > > > > > > > > > ensure > > > > > > > > > > > > no > > > > > > > > > > > > > >> more > > > > > > > > > > > > > >> >> >> > matches can occur. This delay can be > > > > challenging > > > > > > for > > > > > > > > > > > scenarios > > > > > > > > > > > > > that > > > > > > > > > > > > > >> >> >> require > > > > > > > > > > > > > >> >> >> > fast feedback. Early fire support > > addresses > > > > this > > > > > > by > > > > > > > > > > emitting > > > > > > > > > > > > > >> >> intermediate > > > > > > > > > > > > > >> >> >> > results speculatively and using > > retractions > > > or > > > > > > > updates > > > > > > > > > to > > > > > > > > > > > > > maintain > > > > > > > > > > > > > >> >> >> eventual > > > > > > > > > > > > > >> >> >> > consistency and ensure correctness. > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link > > > > > > > > > > > > > >> >> >> > < > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$ > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to > > > refine > > > > > this > > > > > > > > > > feature. > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > Thanks, > > > > > > > > > > > > > >> >> >> > Weiqing > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >