Adding @Yuepeng @Martijn On Mon, Mar 10, 2025 at 7:36 PM Weiqing Yang <[email protected]> wrote:
> Hi Becket, > > Thanks for the feedback! I’ve updated the design doc with a new > “Granularity of the Early Fire Hint” section: link > <https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.gle8q83bjlj5> > . > > Here’s a quick summary: > > - EARLY_FIRE is scoped to a single operator. If multiple operators in > one query can support early firing, Flink applies the hint to the first > matching operator and logs a warning to highlight the ambiguity. Subsequent > operators will not receive the hint. > - Split queries for different early-fire settings. If you need > separate early-fire configurations - or just want to avoid ambiguity - > splitting the SQL into multiple statements or views ensures each operator > has its own distinct EARLY_FIRE hint. > > Please let me know if you have any other questions or suggestions. > > Best, > Weiqing > > On Fri, Feb 28, 2025 at 5:01 PM Becket Qin <[email protected]> wrote: > >> Thanks for updating the FLIP, Weiqing. >> >> Another question, what is the granularity of the early fire hint? Will the >> hint be applied to all the outer joins in the same block? What if in the >> query block there are also aggregations? For example: >> >> SELECT window_start, window_end, buyer, count(*) /*+ >> EARLY_FIRE('delay'='5s' >> , 'time_mode'='rowtime|proctime') */ >> FROM Orders o LEFT JOIN Shipments s >> WHERE o.id = s.order_id >> AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time >> GROUP BY o.window_start, o.window_end, buyer; >> >> What will be the behavior? Will both the aggregation and interval join? Or >> will it only be applied to the join? >> It would be useful to define the behavior here. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <[email protected]> >> wrote: >> >> > Thanks for the feedback, Becket. I’ve heard similar concerns from other >> > reviewers, and I now agree that the `interval` option is likely not >> widely >> > used. In the FLIP, it was intended specifically for Interval Join, not >> as a >> > general early-fire mechanism. We can consider making it more generic in >> a >> > future update. Since it’s optional, I’ll go ahead and remove it for >> > clarity. >> > >> > Appreciate your input! >> > >> > Best, >> > Weiqing >> > >> > On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <[email protected]> >> wrote: >> > >> > > Sorry for the late reply. I have a question regarding the >> configuration >> > of >> > > interval. I am wondering in which scenario the interval config will >> > > actually be used. Is the interval configuration only useful for >> windowed >> > > aggregation? >> > > >> > > For the join operations, early fire seems only applicable to the outer >> > join >> > > before there is a match. Once there is a match, the output of the join >> > > operation should be just event driven. For example, consider the >> > following >> > > scenario: >> > > 1. left outer join, and there is no match from the right side before >> the >> > > initial delay >> > > 2. a record [left, null] was emitted due to early fire >> > > 3. one of the following two cases must happen >> > > a. there is still no match event from the right side before the >> > > specific interval passes. In this case, we are not supposed to emit >> > another >> > > [left, null]. >> > > b. if there is a match, a record of [left, right] should be >> emitted >> > > immediately regardless of the interval. If this happens, the interval >> > will >> > > be ignored from this point on. >> > > >> > > That said, if the configuration proposed in this FLIP is intended not >> > only >> > > for join, but for general purpose early fire, then the interval config >> > > makes sense. >> > > >> > > Thanks, >> > > >> > > Jiangjie (Becket) Qin >> > > >> > > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang < >> [email protected]> >> > > wrote: >> > > >> > > > Thank you all for reviewing. As there are no objections, I’ll move >> > > forward >> > > > with a vote. >> > > > >> > > > Best regards, >> > > > Weiqing >> > > > >> > > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan < >> > > > [email protected]> >> > > > wrote: >> > > > >> > > > > Same here, I don't have any more questions. Thanks! >> > > > > >> > > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <[email protected]> >> > wrote: >> > > > > >> > > > > > Hi Weiqing, >> > > > > > >> > > > > > I don't have any more questions. The doc looks good to me. >> > > > > > >> > > > > > Thanks, >> > > > > > Xingcan >> > > > > > >> > > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan < >> > > > > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Weiqing, >> > > > > > > >> > > > > > > Thanks, that makes sense! Looks like I missed it. >> > > > > > > >> > > > > > > Regards >> > > > > > > Venkata krishnan >> > > > > > > >> > > > > > > >> > > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang < >> > > > > [email protected]> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Venkata, >> > > > > > > > >> > > > > > > > * > where only one earlyFire is fired The DELAY * >> > > > > > > > >> > > > > > > > The DELAY option mentioned in the Public Interfaces section >> > > > > > > > < >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$ >> > > > > > > > > >> > > > > > > > of the proposal can produce a single early fire per >> interval, >> > > > > aligning >> > > > > > > with >> > > > > > > > your suggestion. How it integrates with existing early-fire >> > > > > > > configurations >> > > > > > > > are mentioned here >> > > > > > > > < >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$ >> > > > > > > > >. >> > > > > > > > Let me know if you have any further questions or feedback! >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > Weiqing >> > > > > > > > >> > > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan >> < >> > > > > > > > [email protected]> wrote: >> > > > > > > > >> > > > > > > > > Thanks for the response! >> > > > > > > > > >> > > > > > > > > > If the optional `interval` in the proposal is enabled, >> > > > early-fire >> > > > > > > > outputs >> > > > > > > > > will occur repeatedly at every earlyFireInterval, not just >> > > once. >> > > > > > After >> > > > > > > > that >> > > > > > > > > interval concludes, there will also be a final emission. >> > > > > > > > > >> > > > > > > > > One more question, would it make sense to support a >> mechanism >> > > > > through >> > > > > > > > > configuration where only one earlyFire is fired for an >> > interval >> > > > or >> > > > > > > > window? >> > > > > > > > > This would also simplify the state overhead for use cases >> > where >> > > > > only >> > > > > > > one >> > > > > > > > > early fire is required within a window or interval. >> Thoughts? >> > > > > > > > > >> > > > > > > > > Regards >> > > > > > > > > Venkata krishnan >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hi Venkata, >> > > > > > > > > > >> > > > > > > > > > Thanks for your feedback! >> > > > > > > > > > >> > > > > > > > > > If the optional `interval` in the proposal is enabled, >> > > > early-fire >> > > > > > > > outputs >> > > > > > > > > > will occur repeatedly at every earlyFireInterval, not >> just >> > > > once. >> > > > > > > After >> > > > > > > > > that >> > > > > > > > > > interval concludes, there will also be a final emission. >> > > > > > > > > > >> > > > > > > > > > We haven’t included a late-fire mechanism for interval >> > joins >> > > in >> > > > > > this >> > > > > > > > > FLIP, >> > > > > > > > > > but it’s certainly something we can consider for future >> > > > efforts! >> > > > > > > > > > >> > > > > > > > > > Best regards, >> > > > > > > > > > Weiqing >> > > > > > > > > > >> > > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan >> Sowrirajan >> > < >> > > > > > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Weiqing, >> > > > > > > > > > > >> > > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also >> > instead >> > > > of >> > > > > > > > > > controlling >> > > > > > > > > > > and configuring through Flink configs unlike the older >> > > window >> > > > > > > > > > aggregation, >> > > > > > > > > > > hints seems to be a much better approach. This >> enables a >> > > > > > > customizable >> > > > > > > > > > early >> > > > > > > > > > > fire behavior for individual interval joins. >> > > > > > > > > > > >> > > > > > > > > > > Couple of questions: >> > > > > > > > > > > >> > > > > > > > > > > 1. Does the *early fire* emit an output every >> > > > earlyFireInterval >> > > > > > > time >> > > > > > > > or >> > > > > > > > > > > will it be a one time output emission and another >> output >> > > > > emitted >> > > > > > at >> > > > > > > > the >> > > > > > > > > > end >> > > > > > > > > > > of the interval? >> > > > > > > > > > > 2. Are there plans to support *late fire *similar to >> the >> > > > window >> > > > > > > > > > > aggregations in later FLIPs? >> > > > > > > > > > > >> > > > > > > > > > > Regards >> > > > > > > > > > > Venkata krishnan >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang < >> > > > > > > > [email protected] >> > > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Thanks for reviewing, Xuyang! >> > > > > > > > > > > > >> > > > > > > > > > > > Xingcan (@[email protected]) – do you have any >> > > concerns? >> > > > > > > > > > > > >> > > > > > > > > > > > If no further objections arise from anyone, I’ll >> > proceed >> > > to >> > > > > > mark >> > > > > > > > FLIP >> > > > > > > > > > as >> > > > > > > > > > > > ready for voting. >> > > > > > > > > > > > >> > > > > > > > > > > > Best regards, >> > > > > > > > > > > > Weiqing >> > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang < >> > > [email protected] >> > > > > >> > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > LGTM overall. Thanks for updating. I have no >> problem >> > > and >> > > > +1 >> > > > > > for >> > > > > > > > > this >> > > > > > > > > > > > > feature. >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > -- >> > > > > > > > > > > > > >> > > > > > > > > > > > > Best！ >> > > > > > > > > > > > > Xuyang >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" < >> > > > > > [email protected] >> > > > > > > > >> > > > > > > > > 写道： >> > > > > > > > > > > > > >Hi Xuyang, >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >Thank you for your detailed feedback! I’ve >> updated >> > the >> > > > > > > proposal >> > > > > > > > > doc >> > > > > > > > > > > > > >accordingly. Please feel free to take another >> look >> > and >> > > > let >> > > > > > me >> > > > > > > > know >> > > > > > > > > > if >> > > > > > > > > > > > you >> > > > > > > > > > > > > >have any further thoughts or suggestions. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >Best regards, >> > > > > > > > > > > > > >Weiqing >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang < >> > > > > [email protected]> >> > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> Hi, Weiqing. >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> After reading the new FLIP, I have no issues >> with >> > > the >> > > > > part >> > > > > > > > > `public >> > > > > > > > > > > > > >> interface`. I only have some questions >> regarding >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> the details in the Proposed Changes section. >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the >> > > > > > IntervalJoin >> > > > > > > > > node, >> > > > > > > > > > > > IIUC: >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind >> of >> > > the >> > > > > > > > > IntervalJoin >> > > > > > > > > > > can >> > > > > > > > > > > > > be >> > > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending >> entirely on >> > > the >> > > > > > > > > > requirements >> > > > > > > > > > > of >> > > > > > > > > > > > > the >> > > > > > > > > > > > > >> sink. And the ModifyKind is always ALL. >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> - When early firing is disabled, the >> UpdateKind of >> > > the >> > > > > > > > > > IntervalJoin >> > > > > > > > > > > is >> > > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT. >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> - Nevertheless, whether early firing is >> enabled or >> > > > > > disabled, >> > > > > > > > the >> > > > > > > > > > > > > >> IntervalJoin should always require its input to >> > keep >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with >> > > NONE. >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> -- >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> Best！ >> > > > > > > > > > > > > >> Xuyang >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" < >> > > > > > > > > [email protected]> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> >Hi Xingcan and Xuyang, >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >Thanks so much for the feedback - it was very >> > > > helpful! >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >*> 1. The current output stream of a time >> > interval >> > > > > outer >> > > > > > > join >> > > > > > > > > is >> > > > > > > > > > an >> > > > > > > > > > > > > >> >append-only stream. This change will make it a >> > > > > potential >> > > > > > > > > > > retractable >> > > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a >> > > > dynamic >> > > > > > > output >> > > > > > > > > > type >> > > > > > > > > > > > like >> > > > > > > > > > > > > >> >that. Could you add this part to your design >> > doc?* >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > - Yes, enabling early firing on time >> interval >> > > > outer >> > > > > > > joins >> > > > > > > > > can >> > > > > > > > > > > emit >> > > > > > > > > > > > > >> > retractions when previously emitted rows >> are >> > > > updated >> > > > > > or >> > > > > > > > > > > > invalidated >> > > > > > > > > > > > > by >> > > > > > > > > > > > > >> > later matches. I’ve updated the proposal >> > > (Planner >> > > > > > > > Awareness >> > > > > > > > > > > > > >> > < >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$ >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > and Changes in >> > > FlinkChangelogModeInferenceProgram >> > > > > > > > > > > > > >> > < >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$ >> > > > > > > > > > > > > >> >) >> > > > > > > > > > > > > >> > to clarify that the stream might switch >> from >> > > > > > append-only >> > > > > > > > to >> > > > > > > > > a >> > > > > > > > > > > > > >> > retract/upsert stream. Let me know if >> anything >> > > is >> > > > > > > missing. >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >*> 2. What's the use case when the downstream >> > > > > components >> > > > > > > need >> > > > > > > > > to >> > > > > > > > > > > get >> > > > > > > > > > > > > the >> > > > > > > > > > > > > >> >early fired results regularly?* >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > - The new INTERVAL option (in addition to >> > DELAY) >> > > > > > allows >> > > > > > > > > > periodic >> > > > > > > > > > > > > >> updates >> > > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial >> > > delay. >> > > > > This >> > > > > > > > > > captures >> > > > > > > > > > > > how >> > > > > > > > > > > > > >> results >> > > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s >> > > > > > “Repeatedly” >> > > > > > > > > > option. >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >*> 3. The time interval join operator itself >> is >> > not >> > > > > quite >> > > > > > > > > > efficient >> > > > > > > > > > > > > when >> > > > > > > > > > > > > >> >the state becomes large. Will there be any >> extra >> > > > > overhead >> > > > > > > > after >> > > > > > > > > > > > > >> introducing >> > > > > > > > > > > > > >> >this feature?* >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > - Early fire does introduce some overhead >> by >> > > > > > potentially >> > > > > > > > > > > emitting >> > > > > > > > > > > > > >> > partial matches multiple times with >> retraction >> > > > > > (avoiding >> > > > > > > > > > > duplicate >> > > > > > > > > > > > > >> outputs >> > > > > > > > > > > > > >> > though). However, if it’s disabled, there >> is >> > no >> > > > > > > additional >> > > > > > > > > > cost. >> > > > > > > > > > > > > Most >> > > > > > > > > > > > > >> users >> > > > > > > > > > > > > >> > find the performance trade-off acceptable >> for >> > > the >> > > > > > > > real-time >> > > > > > > > > > > > > insights it >> > > > > > > > > > > > > >> > provides. >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >*> 1. Currently, there are some configs >> related >> > to >> > > > > early >> > > > > > > > firing >> > > > > > > > > > > > > available >> > > > > > > > > > > > > >> >to users: >> `table.exec.emit.early-fire.en**abled` >> > > and >> > > > > > > > > > > > > >> >`table.exec.emit.early-fire.de < >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$ >> > > > > > > > > > > > > >> >**lay`. >> > > > > > > > > > > > > >> >Although their documentation states that they >> are >> > > > only >> > > > > > > > > applicable >> > > > > > > > > > > to >> > > > > > > > > > > > > the >> > > > > > > > > > > > > >> >Window operator, it seems possible that they >> may >> > > also >> > > > > be >> > > > > > > > > relevant >> > > > > > > > > > > in >> > > > > > > > > > > > > the >> > > > > > > > > > > > > >> >context of this FLIP. Otherwise, having >> different >> > > > early >> > > > > > > > firing >> > > > > > > > > > > > > behaviors >> > > > > > > > > > > > > >> >for different operators could confuse users.* >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to >> avoid >> > > > > > > confusion. >> > > > > > > > > I’ve >> > > > > > > > > > > > > added a >> > > > > > > > > > > > > >> > section >> > > > > > > > > > > > > >> > < >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$ >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >in >> > > > > > > > > > > > > >> > the proposal highlighting that we should >> align >> > > > > > > hint-based >> > > > > > > > > > > interval >> > > > > > > > > > > > > join >> > > > > > > > > > > > > >> > configurations with the existing >> > > table.exec.emit.* >> > > > > > > > settings. >> > > > > > > > > > > > > >> Suggestions >> > > > > > > > > > > > > >> > on how to make the unification are >> welcome! We >> > > > plan >> > > > > to >> > > > > > > > > extend >> > > > > > > > > > > > early >> > > > > > > > > > > > > >> firing >> > > > > > > > > > > > > >> > to window joins via hints in a future FLIP. >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent. >> > > Similar >> > > > > to >> > > > > > > > point >> > > > > > > > > 1, >> > > > > > > > > > > > > perhaps >> > > > > > > > > > > > > >> >we can introduce it to other window-related >> > > operators >> > > > > in >> > > > > > > the >> > > > > > > > > > > future.> >> > > > > > > > > > > > > 3. >> > > > > > > > > > > > > >> >You need to modify the >> > > > > FlinkChangelogModeInferenceProgram >> > > > > > > to >> > > > > > > > > > ensure >> > > > > > > > > > > > > that >> > > > > > > > > > > > > >> >downstream nodes of interval joins with early >> > > firing >> > > > > > > enabled >> > > > > > > > > are >> > > > > > > > > > > > aware >> > > > > > > > > > > > > of >> > > > > > > > > > > > > >> >retract or upsert messages.* >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > - We agree that time_mode could be >> introduced >> > to >> > > > > other >> > > > > > > > > > > > window-based >> > > > > > > > > > > > > >> > operators down the road. We also want to >> > support >> > > > > early >> > > > > > > > fire >> > > > > > > > > > for >> > > > > > > > > > > > > >> > window join. Also, thanks for highlighting >> > > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added >> > the >> > > > code >> > > > > > > > change >> > > > > > > > > on >> > > > > > > > > > > it >> > > > > > > > > > > > > >> > < >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$ >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > in the proposal. >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve >> > > updated >> > > > > the >> > > > > > > > > > proposal >> > > > > > > > > > > > with >> > > > > > > > > > > > > >> >these points. Please let me know if there’s >> > > anything >> > > > > > else I >> > > > > > > > > > should >> > > > > > > > > > > > > >> address. >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >Best, >> > > > > > > > > > > > > >> >Weiqing >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang < >> > > > > > [email protected]> >> > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this >> FLIP. >> > I >> > > > > have a >> > > > > > > few >> > > > > > > > > > > > > questions: >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> 1. Currently, there are some configs >> related to >> > > > early >> > > > > > > > firing >> > > > > > > > > > > > > available >> > > > > > > > > > > > > >> to >> > > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` >> and >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although >> > > their >> > > > > > > > > > documentation >> > > > > > > > > > > > > states >> > > > > > > > > > > > > >> >> that they are only applicable to the Window >> > > > operator, >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> it seems possible that they may also be >> > relevant >> > > in >> > > > > the >> > > > > > > > > context >> > > > > > > > > > > of >> > > > > > > > > > > > > this >> > > > > > > > > > > > > >> >> FLIP. Otherwise, having different early >> firing >> > > > > > behaviors >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> for different operators could confuse users. >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent. >> > > Similar >> > > > to >> > > > > > > point >> > > > > > > > > 1, >> > > > > > > > > > > > > perhaps >> > > > > > > > > > > > > >> we >> > > > > > > > > > > > > >> >> can introduce it to other window-related >> > > operators >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> in the future. >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> 3. You need to modify the >> > > > > > > > FlinkChangelogModeInferenceProgram >> > > > > > > > > to >> > > > > > > > > > > > > ensure >> > > > > > > > > > > > > >> >> that downstream nodes of interval joins with >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> early firing enabled are aware of retract or >> > > upsert >> > > > > > > > messages. >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> -- >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> Best！ >> > > > > > > > > > > > > >> >> Xuyang >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" < >> > > > > > > [email protected]> >> > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> >> >Hi Weiqing, >> > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early >> > fire >> > > > for >> > > > > > time >> > > > > > > > > > > interval >> > > > > > > > > > > > > outer >> > > > > > > > > > > > > >> >> >joins is feasible overall. I have a few >> > > questions. >> > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > >> >> >1. The current output stream of a time >> > interval >> > > > > outer >> > > > > > > join >> > > > > > > > > is >> > > > > > > > > > an >> > > > > > > > > > > > > >> >> >append-only stream. This change will make >> it a >> > > > > > potential >> > > > > > > > > > > > retractable >> > > > > > > > > > > > > >> >> >stream. I'm not sure if the planner >> supports a >> > > > > dynamic >> > > > > > > > > output >> > > > > > > > > > > type >> > > > > > > > > > > > > like >> > > > > > > > > > > > > >> >> >that. Could you add this part to your >> design >> > > doc? >> > > > > > > > > > > > > >> >> >2. What's the use case when the downstream >> > > > > components >> > > > > > > need >> > > > > > > > > to >> > > > > > > > > > > get >> > > > > > > > > > > > > the >> > > > > > > > > > > > > >> >> early >> > > > > > > > > > > > > >> >> >fired results regularly? >> > > > > > > > > > > > > >> >> >3. The time interval join operator itself >> is >> > not >> > > > > quite >> > > > > > > > > > efficient >> > > > > > > > > > > > > when >> > > > > > > > > > > > > >> the >> > > > > > > > > > > > > >> >> >state becomes large. Will there be any >> extra >> > > > > overhead >> > > > > > > > after >> > > > > > > > > > > > > introducing >> > > > > > > > > > > > > >> >> >this feature? >> > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > >> >> >Thanks, >> > > > > > > > > > > > > >> >> >Xingcan >> > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing >> Yang < >> > > > > > > > > > > > > [email protected]> >> > > > > > > > > > > > > >> >> >wrote: >> > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > >> >> >> Hi all, >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the >> > proposal >> > > I >> > > > > > shared >> > > > > > > > on >> > > > > > > > > > > early >> > > > > > > > > > > > > fire >> > > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d >> > > > greatly >> > > > > > > > > appreciate >> > > > > > > > > > > > your >> > > > > > > > > > > > > >> >> feedback >> > > > > > > > > > > > > >> >> >> or suggestions. >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> Here’s the link to the proposal document: >> > Link >> > > > > > > > > > > > > >> >> >> < >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$ >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> Thank you! >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> Best, >> > > > > > > > > > > > > >> >> >> Weiqing >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing >> > Yang >> > > < >> > > > > > > > > > > > > >> [email protected] >> > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > >> >> >> wrote: >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> > Hi all, >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about >> > > > > > introducing >> > > > > > > > > early >> > > > > > > > > > > fire >> > > > > > > > > > > > > >> support >> > > > > > > > > > > > > >> >> >> for >> > > > > > > > > > > > > >> >> >> > Flink SQL interval joins. >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> > *Motivation* >> > > > > > > > > > > > > >> >> >> > In many streaming applications, >> > particularly >> > > > > > > real-time >> > > > > > > > > > > > analytics >> > > > > > > > > > > > > >> and >> > > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to >> > obtain >> > > > > > partial >> > > > > > > > > > results >> > > > > > > > > > > > > >> earlier >> > > > > > > > > > > > > >> >> >> rather >> > > > > > > > > > > > > >> >> >> > than waiting for full join conditions >> to >> > > > > finalize. >> > > > > > > For >> > > > > > > > > > Flink >> > > > > > > > > > > > SQL >> > > > > > > > > > > > > >> >> interval >> > > > > > > > > > > > > >> >> >> > joins, results are typically delayed >> until >> > > > > > > watermarks >> > > > > > > > > > ensure >> > > > > > > > > > > > no >> > > > > > > > > > > > > >> more >> > > > > > > > > > > > > >> >> >> > matches can occur. This delay can be >> > > > challenging >> > > > > > for >> > > > > > > > > > > scenarios >> > > > > > > > > > > > > that >> > > > > > > > > > > > > >> >> >> require >> > > > > > > > > > > > > >> >> >> > fast feedback. Early fire support >> > addresses >> > > > this >> > > > > > by >> > > > > > > > > > emitting >> > > > > > > > > > > > > >> >> intermediate >> > > > > > > > > > > > > >> >> >> > results speculatively and using >> > retractions >> > > or >> > > > > > > updates >> > > > > > > > > to >> > > > > > > > > > > > > maintain >> > > > > > > > > > > > > >> >> >> eventual >> > > > > > > > > > > > > >> >> >> > consistency and ensure correctness. >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link >> > > > > > > > > > > > > >> >> >> > < >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$ >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to >> > > refine >> > > > > this >> > > > > > > > > > feature. >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> > Thanks, >> > > > > > > > > > > > > >> >> >> > Weiqing >> > > > > > > > > > > > > >> >> >> > >> > > > > > > > > > > > > >> >> >> >> > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > >> >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
