Thanks for the response! > If the optional `interval` in the proposal is enabled, early-fire outputs will occur repeatedly at every earlyFireInterval, not just once. After that interval concludes, there will also be a final emission.
One more question, would it make sense to support a mechanism through configuration where only one earlyFire is fired for an interval or window? This would also simplify the state overhead for use cases where only one early fire is required within a window or interval. Thoughts? Regards Venkata krishnan On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <yangweiqing...@gmail.com> wrote: > Hi Venkata, > > Thanks for your feedback! > > If the optional `interval` in the proposal is enabled, early-fire outputs > will occur repeatedly at every earlyFireInterval, not just once. After that > interval concludes, there will also be a final emission. > > We haven’t included a late-fire mechanism for interval joins in this FLIP, > but it’s certainly something we can consider for future efforts! > > Best regards, > Weiqing > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> > wrote: > > > Weiqing, > > > > Thanks, this is a great addition to Flink SQL. Also instead of > controlling > > and configuring through Flink configs unlike the older window > aggregation, > > hints seems to be a much better approach. This enables a customizable > early > > fire behavior for individual interval joins. > > > > Couple of questions: > > > > 1. Does the *early fire* emit an output every earlyFireInterval time or > > will it be a one time output emission and another output emitted at the > end > > of the interval? > > 2. Are there plans to support *late fire *similar to the window > > aggregations in later FLIPs? > > > > Regards > > Venkata krishnan > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <yangweiqing...@gmail.com> > > wrote: > > > > > Thanks for reviewing, Xuyang! > > > > > > Xingcan (@xingc...@gmail.com) – do you have any concerns? > > > > > > If no further objections arise from anyone, I’ll proceed to mark FLIP > as > > > ready for voting. > > > > > > Best regards, > > > Weiqing > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xyzhong...@163.com> wrote: > > > > > > > LGTM overall. Thanks for updating. I have no problem and +1 for this > > > > feature. > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Best！ > > > > Xuyang > > > > > > > > > > > > > > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <yangweiqing...@gmail.com> 写道： > > > > >Hi Xuyang, > > > > > > > > > >Thank you for your detailed feedback! I’ve updated the proposal doc > > > > >accordingly. Please feel free to take another look and let me know > if > > > you > > > > >have any further thoughts or suggestions. > > > > > > > > > >Best regards, > > > > >Weiqing > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <xyzhong...@163.com> wrote: > > > > > > > > > >> Hi, Weiqing. > > > > >> > > > > >> After reading the new FLIP, I have no issues with the part `public > > > > >> interface`. I only have some questions regarding > > > > >> > > > > >> the details in the Proposed Changes section. > > > > >> > > > > >> Regarding the ModifyKind and UpdateKind of the IntervalJoin node, > > > IIUC: > > > > >> > > > > >> - When early firing is enabled, the UpdateKind of the IntervalJoin > > can > > > > be > > > > >> either ONLY_UPDATE_AFTER or > > > > >> > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the > requirements > > of > > > > the > > > > >> sink. And the ModifyKind is always ALL. > > > > >> > > > > >> - When early firing is disabled, the UpdateKind of the > IntervalJoin > > is > > > > >> NONE, and the ModifyKind is INSERT. > > > > >> > > > > >> - Nevertheless, whether early firing is enabled or disabled, the > > > > >> IntervalJoin should always require its input to keep > > > > >> > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE. > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Best！ > > > > >> Xuyang > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <yangweiqing...@gmail.com> > > > > wrote: > > > > >> >Hi Xingcan and Xuyang, > > > > >> > > > > > >> >Thanks so much for the feedback - it was very helpful! > > > > >> > > > > > >> >*> 1. The current output stream of a time interval outer join is > an > > > > >> >append-only stream. This change will make it a potential > > retractable > > > > >> >stream. I'm not sure if the planner supports a dynamic output > type > > > like > > > > >> >that. Could you add this part to your design doc?* > > > > >> > > > > > >> > > > > > >> > - Yes, enabling early firing on time interval outer joins can > > emit > > > > >> > retractions when previously emitted rows are updated or > > > invalidated > > > > by > > > > >> > later matches. I’ve updated the proposal (Planner Awareness > > > > >> > < > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$ > > > > >> > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram > > > > >> > < > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$ > > > > >> >) > > > > >> > to clarify that the stream might switch from append-only to a > > > > >> > retract/upsert stream. Let me know if anything is missing. > > > > >> > > > > > >> > > > > > >> >*> 2. What's the use case when the downstream components need to > > get > > > > the > > > > >> >early fired results regularly?* > > > > >> > > > > > >> > > > > > >> > - The new INTERVAL option (in addition to DELAY) allows > periodic > > > > >> updates > > > > >> > (e.g., every 10 minutes) after the initial delay. This > captures > > > how > > > > >> results > > > > >> > evolve over time, similar to Apache Beam’s “Repeatedly” > option. > > > > >> > > > > > >> > > > > > >> >*> 3. The time interval join operator itself is not quite > efficient > > > > when > > > > >> >the state becomes large. Will there be any extra overhead after > > > > >> introducing > > > > >> >this feature?* > > > > >> > > > > > >> > - Early fire does introduce some overhead by potentially > > emitting > > > > >> > partial matches multiple times with retraction (avoiding > > duplicate > > > > >> outputs > > > > >> > though). However, if it’s disabled, there is no additional > cost. > > > > Most > > > > >> users > > > > >> > find the performance trade-off acceptable for the real-time > > > > insights it > > > > >> > provides. > > > > >> > > > > > >> > > > > > >> >*> 1. Currently, there are some configs related to early firing > > > > available > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and > > > > >> >`table.exec.emit.early-fire.de < > > > > > > https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$ > > > > >> >**lay`. > > > > >> >Although their documentation states that they are only applicable > > to > > > > the > > > > >> >Window operator, it seems possible that they may also be relevant > > in > > > > the > > > > >> >context of this FLIP. Otherwise, having different early firing > > > > behaviors > > > > >> >for different operators could confuse users.* > > > > >> > > > > > >> > - +1 on unifying early-fire behaviors to avoid confusion. I’ve > > > > added a > > > > >> > section > > > > >> > < > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$ > > > > >> > > > > > >> >in > > > > >> > the proposal highlighting that we should align hint-based > > interval > > > > join > > > > >> > configurations with the existing table.exec.emit.* settings. > > > > >> Suggestions > > > > >> > on how to make the unification are welcome! We plan to extend > > > early > > > > >> firing > > > > >> > to window joins via hints in a future FLIP. > > > > >> > > > > > >> > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar to point 1, > > > > perhaps > > > > >> >we can introduce it to other window-related operators in the > > future.> > > > > 3. > > > > >> >You need to modify the FlinkChangelogModeInferenceProgram to > ensure > > > > that > > > > >> >downstream nodes of interval joins with early firing enabled are > > > aware > > > > of > > > > >> >retract or upsert messages.* > > > > >> > > > > > >> > - We agree that time_mode could be introduced to other > > > window-based > > > > >> > operators down the road. We also want to support early fire > for > > > > >> > window join. Also, thanks for highlighting > > > > >> > FlinkChangelogModeInferenceProgram! I added the code change on > > it > > > > >> > < > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$ > > > > >> > > > > > >> > in the proposal. > > > > >> > > > > > >> > > > > > >> >Thanks again for your time and feedback! I’ve updated the > proposal > > > with > > > > >> >these points. Please let me know if there’s anything else I > should > > > > >> address. > > > > >> > > > > > >> >Best, > > > > >> >Weiqing > > > > >> > > > > > >> > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <xyzhong...@163.com> > wrote: > > > > >> > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I have a few > > > > questions: > > > > >> >> > > > > >> >> 1. Currently, there are some configs related to early firing > > > > available > > > > >> to > > > > >> >> users: `table.exec.emit.early-fire.enabled` and > > > > >> >> > > > > >> >> `table.exec.emit.early-fire.delay`. Although their > documentation > > > > states > > > > >> >> that they are only applicable to the Window operator, > > > > >> >> > > > > >> >> it seems possible that they may also be relevant in the context > > of > > > > this > > > > >> >> FLIP. Otherwise, having different early firing behaviors > > > > >> >> > > > > >> >> for different operators could confuse users. > > > > >> >> > > > > >> >> 2. The design of `time_mode` is excellent. Similar to point 1, > > > > perhaps > > > > >> we > > > > >> >> can introduce it to other window-related operators > > > > >> >> > > > > >> >> in the future. > > > > >> >> > > > > >> >> 3. You need to modify the FlinkChangelogModeInferenceProgram to > > > > ensure > > > > >> >> that downstream nodes of interval joins with > > > > >> >> > > > > >> >> early firing enabled are aware of retract or upsert messages. > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> -- > > > > >> >> > > > > >> >> Best！ > > > > >> >> Xuyang > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <xingc...@gmail.com> > > wrote: > > > > >> >> >Hi Weiqing, > > > > >> >> > > > > > >> >> >Thanks for the proposal. IMO, adding early fire for time > > interval > > > > outer > > > > >> >> >joins is feasible overall. I have a few questions. > > > > >> >> > > > > > >> >> >1. The current output stream of a time interval outer join is > an > > > > >> >> >append-only stream. This change will make it a potential > > > retractable > > > > >> >> >stream. I'm not sure if the planner supports a dynamic output > > type > > > > like > > > > >> >> >that. Could you add this part to your design doc? > > > > >> >> >2. What's the use case when the downstream components need to > > get > > > > the > > > > >> >> early > > > > >> >> >fired results regularly? > > > > >> >> >3. The time interval join operator itself is not quite > efficient > > > > when > > > > >> the > > > > >> >> >state becomes large. Will there be any extra overhead after > > > > introducing > > > > >> >> >this feature? > > > > >> >> > > > > > >> >> >Thanks, > > > > >> >> >Xingcan > > > > >> >> > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang < > > > > yangweiqing...@gmail.com> > > > > >> >> >wrote: > > > > >> >> > > > > > >> >> >> Hi all, > > > > >> >> >> > > > > >> >> >> Just a gentle reminder regarding the proposal I shared on > > early > > > > fire > > > > >> >> >> support for Flink SQL interval joins. I’d greatly appreciate > > > your > > > > >> >> feedback > > > > >> >> >> or suggestions. > > > > >> >> >> > > > > >> >> >> Here’s the link to the proposal document: Link > > > > >> >> >> < > > > > >> >> >> > > > > >> >> > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$ > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> Thank you! > > > > >> >> >> > > > > >> >> >> Best, > > > > >> >> >> Weiqing > > > > >> >> >> > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang < > > > > >> yangweiqing...@gmail.com > > > > >> >> > > > > > >> >> >> wrote: > > > > >> >> >> > > > > >> >> >> > Hi all, > > > > >> >> >> > > > > > >> >> >> > I’d like to initiate a discussion about introducing early > > fire > > > > >> support > > > > >> >> >> for > > > > >> >> >> > Flink SQL interval joins. > > > > >> >> >> > > > > > >> >> >> > *Motivation* > > > > >> >> >> > In many streaming applications, particularly real-time > > > analytics > > > > >> and > > > > >> >> >> > monitoring systems, it is valuable to obtain partial > results > > > > >> earlier > > > > >> >> >> rather > > > > >> >> >> > than waiting for full join conditions to finalize. For > Flink > > > SQL > > > > >> >> interval > > > > >> >> >> > joins, results are typically delayed until watermarks > ensure > > > no > > > > >> more > > > > >> >> >> > matches can occur. This delay can be challenging for > > scenarios > > > > that > > > > >> >> >> require > > > > >> >> >> > fast feedback. Early fire support addresses this by > emitting > > > > >> >> intermediate > > > > >> >> >> > results speculatively and using retractions or updates to > > > > maintain > > > > >> >> >> eventual > > > > >> >> >> > consistency and ensure correctness. > > > > >> >> >> > > > > > >> >> >> > Here’s the proposal document: Link > > > > >> >> >> > < > > > > >> >> >> > > > > >> >> > > > > >> > > > > > > > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$ > > > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> > Your feedback and ideas are welcome to refine this > feature. > > > > >> >> >> > > > > > >> >> >> > Thanks, > > > > >> >> >> > Weiqing > > > > >> >> >> > > > > > >> >> >> > > > > >> >> > > > > >> > > > > > > > > > >
