Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Becket, Thanks for the feedback! I’ve updated the design doc with a new “Granularity of the Early Fire Hint” section: link . Here’s a quick summary: - EARLY_FIRE is scoped

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Thanks for updating the FLIP, Weiqing. Another question, what is the granularity of the early fire hint? Will the hint be applied to all the outer joins in the same block? What if in the query block there are also aggregations? For example: SELECT window_start, window_end, buyer, count(*) /*+ EAR

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Thanks for the feedback, Becket. I’ve heard similar concerns from other reviewers, and I now agree that the `interval` option is likely not widely used. In the FLIP, it was intended specifically for Interval Join, not as a general early-fire mechanism. We can consider making it more generic in a fu

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Sorry for the late reply. I have a question regarding the configuration of interval. I am wondering in which scenario the interval config will actually be used. Is the interval configuration only useful for windowed aggregation? For the join operations, early fire seems only applicable to the oute

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Thank you all for reviewing. As there are no objections, I’ll move forward with a vote. Best regards, Weiqing On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan wrote: > Same here, I don't have any more questions. Thanks! > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui wrote: > > > Hi W

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Same here, I don't have any more questions. Thanks! On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui wrote: > Hi Weiqing, > > I don't have any more questions. The doc looks good to me. > > Thanks, > Xingcan > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> > wrote: >

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Weiqing, I don't have any more questions. The doc looks good to me. Thanks, Xingcan On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan wrote: > Hi Weiqing, > > Thanks, that makes sense! Looks like I missed it. > > Regards > Venkata krishnan > > > On Tue, Jan 21, 2025 at 10:55 PM Wei

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Weiqing, Thanks, that makes sense! Looks like I missed it. Regards Venkata krishnan On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang wrote: > Hi Venkata, > > * > where only one earlyFire is fired The DELAY * > > The DELAY option mentioned in the Public Interfaces section > < > https://urldefe

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Venkata, * > where only one earlyFire is fired The DELAY * The DELAY option mentioned in the Public Interfaces section of the proposal can produce a single early fire per inter

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Thanks for the response! > If the optional `interval` in the proposal is enabled, early-fire outputs will occur repeatedly at every earlyFireInterval, not just once. After that interval concludes, there will also be a final emission. One more question, would it make sense to support a mechanism t

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Venkata, Thanks for your feedback! If the optional `interval` in the proposal is enabled, early-fire outputs will occur repeatedly at every earlyFireInterval, not just once. After that interval concludes, there will also be a final emission. We haven’t included a late-fire mechanism for inter

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Weiqing, Thanks, this is a great addition to Flink SQL. Also instead of controlling and configuring through Flink configs unlike the older window aggregation, hints seems to be a much better approach. This enables a customizable early fire behavior for individual interval joins. Couple of questio

Re: Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Thanks for reviewing, Xuyang! Xingcan (@xingc...@gmail.com) – do you have any concerns? If no further objections arise from anyone, I’ll proceed to mark FLIP as ready for voting. Best regards, Weiqing On Tue, Jan 14, 2025 at 9:06 PM Xuyang wrote: > LGTM overall. Thanks for updating. I have no

Re:Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

LGTM overall. Thanks for updating. I have no problem and +1 for this feature. -- Best！ Xuyang 在 2025-01-15 12:33:16，"Weiqing Yang" 写道： >Hi Xuyang, > >Thank you for your detailed feedback! I’ve updated the proposal doc >accordingly. Please feel free to take another look and let me

Re: Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Xuyang, Thank you for your detailed feedback! I’ve updated the proposal doc accordingly. Please feel free to take another look and let me know if you have any further thoughts or suggestions. Best regards, Weiqing On Mon, Jan 13, 2025 at 3:50 AM Xuyang wrote: > Hi, Weiqing. > > After readin

Re:Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi, Weiqing. After reading the new FLIP, I have no issues with the part `public interface`. I only have some questions regarding the details in the Proposed Changes section. Regarding the ModifyKind and UpdateKind of the IntervalJoin node, IIUC: - When early firing is enabled, the UpdateKin

Re: Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Xingcan and Xuyang, Thanks so much for the feedback - it was very helpful! *> 1. The current output stream of a time interval outer join is an append-only stream. This change will make it a potential retractable stream. I'm not sure if the planner supports a dynamic output type like that. Coul

Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi Weiqing, Thanks for the proposal. IMO, adding early fire for time interval outer joins is feasible overall. I have a few questions. 1. The current output stream of a time interval outer join is an append-only stream. This change will make it a potential retractable stream. I'm not sure if the

Re: [DISCUSS] Early Fire Support for Flink SQL Interval Join

Hi all, Just a gentle reminder regarding the proposal I shared on early fire support for Flink SQL interval joins. I’d greatly appreciate your feedback or suggestions. Here’s the link to the proposal document: Link