Sorry for the late reply. I have a question regarding the configuration of
interval. I am wondering in which scenario the interval config will
actually be used. Is the interval configuration only useful for windowed
aggregation?
For the join operations, early fire seems only applicable to the outer join
before there is a match. Once there is a match, the output of the join
operation should be just event driven. For example, consider the following
scenario:
1. left outer join, and there is no match from the right side before the
initial delay
2. a record [left, null] was emitted due to early fire
3. one of the following two cases must happen
a. there is still no match event from the right side before the
specific interval passes. In this case, we are not supposed to emit another
[left, null].
b. if there is a match, a record of [left, right] should be emitted
immediately regardless of the interval. If this happens, the interval will
be ignored from this point on.
That said, if the configuration proposed in this FLIP is intended not only
for join, but for general purpose early fire, then the interval config
makes sense.
Thanks,
Jiangjie (Becket) Qin
On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <yangweiqing...@gmail.com>
wrote:
> Thank you all for reviewing. As there are no objections, I’ll move forward
> with a vote.
>
> Best regards,
> Weiqing
>
> On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu>
> wrote:
>
> > Same here, I don't have any more questions. Thanks!
> >
> > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xingc...@gmail.com> wrote:
> >
> > > Hi Weiqing,
> > >
> > > I don't have any more questions. The doc looks good to me.
> > >
> > > Thanks,
> > > Xingcan
> > >
> > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > vsowr...@asu.edu>
> > > wrote:
> > >
> > > > Hi Weiqing,
> > > >
> > > > Thanks, that makes sense! Looks like I missed it.
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > >
> > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > yangweiqing...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Venkata,
> > > > >
> > > > > * > where only one earlyFire is fired The DELAY *
> > > > >
> > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > <
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > >
> > > > > of the proposal can produce a single early fire per interval,
> > aligning
> > > > with
> > > > > your suggestion. How it integrates with existing early-fire
> > > > configurations
> > > > > are mentioned here
> > > > > <
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > >.
> > > > > Let me know if you have any further questions or feedback!
> > > > >
> > > > > Thanks,
> > > > > Weiqing
> > > > >
> > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > vsowr...@asu.edu> wrote:
> > > > >
> > > > > > Thanks for the response!
> > > > > >
> > > > > > > If the optional `interval` in the proposal is enabled,
> early-fire
> > > > > outputs
> > > > > > will occur repeatedly at every earlyFireInterval, not just once.
> > > After
> > > > > that
> > > > > > interval concludes, there will also be a final emission.
> > > > > >
> > > > > > One more question, would it make sense to support a mechanism
> > through
> > > > > > configuration where only one earlyFire is fired for an interval
> or
> > > > > window?
> > > > > > This would also simplify the state overhead for use cases where
> > only
> > > > one
> > > > > > early fire is required within a window or interval. Thoughts?
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > yangweiqing...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Venkata,
> > > > > > >
> > > > > > > Thanks for your feedback!
> > > > > > >
> > > > > > > If the optional `interval` in the proposal is enabled,
> early-fire
> > > > > outputs
> > > > > > > will occur repeatedly at every earlyFireInterval, not just
> once.
> > > > After
> > > > > > that
> > > > > > > interval concludes, there will also be a final emission.
> > > > > > >
> > > > > > > We haven’t included a late-fire mechanism for interval joins in
> > > this
> > > > > > FLIP,
> > > > > > > but it’s certainly something we can consider for future
> efforts!
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Weiqing
> > > > > > >
> > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan <
> > > > > > > vsowr...@asu.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Weiqing,
> > > > > > > >
> > > > > > > > Thanks, this is a great addition to Flink SQL. Also instead
> of
> > > > > > > controlling
> > > > > > > > and configuring through Flink configs unlike the older window
> > > > > > > aggregation,
> > > > > > > > hints seems to be a much better approach. This enables a
> > > > customizable
> > > > > > > early
> > > > > > > > fire behavior for individual interval joins.
> > > > > > > >
> > > > > > > > Couple of questions:
> > > > > > > >
> > > > > > > > 1. Does the *early fire* emit an output every
> earlyFireInterval
> > > > time
> > > > > or
> > > > > > > > will it be a one time output emission and another output
> > emitted
> > > at
> > > > > the
> > > > > > > end
> > > > > > > > of the interval?
> > > > > > > > 2. Are there plans to support *late fire *similar to the
> window
> > > > > > > > aggregations in later FLIPs?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Venkata krishnan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > yangweiqing...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > >
> > > > > > > > > Xingcan (@xingc...@gmail.com) – do you have any concerns?
> > > > > > > > >
> > > > > > > > > If no further objections arise from anyone, I’ll proceed to
> > > mark
> > > > > FLIP
> > > > > > > as
> > > > > > > > > ready for voting.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Weiqing
> > > > > > > > >
> > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <xyzhong...@163.com
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > LGTM overall. Thanks for updating. I have no problem and
> +1
> > > for
> > > > > > this
> > > > > > > > > > feature.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Best！
> > > > > > > > > > Xuyang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > yangweiqing...@gmail.com
> > > > >
> > > > > > 写道：
> > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > >
> > > > > > > > > > >Thank you for your detailed feedback! I’ve updated the
> > > > proposal
> > > > > > doc
> > > > > > > > > > >accordingly. Please feel free to take another look and
> let
> > > me
> > > > > know
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > >
> > > > > > > > > > >Best regards,
> > > > > > > > > > >Weiqing
> > > > > > > > > > >
> > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > xyzhong...@163.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > >>
> > > > > > > > > > >> After reading the new FLIP, I have no issues with the
> > part
> > > > > > `public
> > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > >>
> > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > >>
> > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > IntervalJoin
> > > > > > node,
> > > > > > > > > IIUC:
> > > > > > > > > > >>
> > > > > > > > > > >> - When early firing is enabled, the UpdateKind of the
> > > > > > IntervalJoin
> > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > >>
> > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on the
> > > > > > > requirements
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > >>
> > > > > > > > > > >> - When early firing is disabled, the UpdateKind of the
> > > > > > > IntervalJoin
> > > > > > > > is
> > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > >>
> > > > > > > > > > >> - Nevertheless, whether early firing is enabled or
> > > disabled,
> > > > > the
> > > > > > > > > > >> IntervalJoin should always require its input to keep
> > > > > > > > > > >>
> > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with NONE.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >>
> > > > > > > > > > >> Best！
> > > > > > > > > > >> Xuyang
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > yangweiqing...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > >> >
> > > > > > > > > > >> >Thanks so much for the feedback - it was very
> helpful!
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 1. The current output stream of a time interval
> > outer
> > > > join
> > > > > > is
> > > > > > > an
> > > > > > > > > > >> >append-only stream. This change will make it a
> > potential
> > > > > > > > retractable
> > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> dynamic
> > > > output
> > > > > > > type
> > > > > > > > > like
> > > > > > > > > > >> >that. Could you add this part to your design doc?*
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > - Yes, enabling early firing on time interval
> outer
> > > > joins
> > > > > > can
> > > > > > > > emit
> > > > > > > > > > >> > retractions when previously emitted rows are
> updated
> > > or
> > > > > > > > > invalidated
> > > > > > > > > > by
> > > > > > > > > > >> > later matches. I’ve updated the proposal (Planner
> > > > > Awareness
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > >> >
> > > > > > > > > > >> > and Changes in FlinkChangelogModeInferenceProgram
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > >> >)
> > > > > > > > > > >> > to clarify that the stream might switch from
> > > append-only
> > > > > to
> > > > > > a
> > > > > > > > > > >> > retract/upsert stream. Let me know if anything is
> > > > missing.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > components
> > > > need
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > - The new INTERVAL option (in addition to DELAY)
> > > allows
> > > > > > > periodic
> > > > > > > > > > >> updates
> > > > > > > > > > >> > (e.g., every 10 minutes) after the initial delay.
> > This
> > > > > > > captures
> > > > > > > > > how
> > > > > > > > > > >> results
> > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > “Repeatedly”
> > > > > > > option.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 3. The time interval join operator itself is not
> > quite
> > > > > > > efficient
> > > > > > > > > > when
> > > > > > > > > > >> >the state becomes large. Will there be any extra
> > overhead
> > > > > after
> > > > > > > > > > >> introducing
> > > > > > > > > > >> >this feature?*
> > > > > > > > > > >> >
> > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > potentially
> > > > > > > > emitting
> > > > > > > > > > >> > partial matches multiple times with retraction
> > > (avoiding
> > > > > > > > duplicate
> > > > > > > > > > >> outputs
> > > > > > > > > > >> > though). However, if it’s disabled, there is no
> > > > additional
> > > > > > > cost.
> > > > > > > > > > Most
> > > > > > > > > > >> users
> > > > > > > > > > >> > find the performance trade-off acceptable for the
> > > > > real-time
> > > > > > > > > > insights it
> > > > > > > > > > >> > provides.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 1. Currently, there are some configs related to
> > early
> > > > > firing
> > > > > > > > > > available
> > > > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled` and
> > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > >> >**lay`.
> > > > > > > > > > >> >Although their documentation states that they are
> only
> > > > > > applicable
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > >> >Window operator, it seems possible that they may also
> > be
> > > > > > relevant
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > >> >context of this FLIP. Otherwise, having different
> early
> > > > > firing
> > > > > > > > > > behaviors
> > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > >> >
> > > > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > > > confusion.
> > > > > > I’ve
> > > > > > > > > > added a
> > > > > > > > > > >> > section
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > >> >
> > > > > > > > > > >> >in
> > > > > > > > > > >> > the proposal highlighting that we should align
> > > > hint-based
> > > > > > > > interval
> > > > > > > > > > join
> > > > > > > > > > >> > configurations with the existing table.exec.emit.*
> > > > > settings.
> > > > > > > > > > >> Suggestions
> > > > > > > > > > >> > on how to make the unification are welcome! We
> plan
> > to
> > > > > > extend
> > > > > > > > > early
> > > > > > > > > > >> firing
> > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent. Similar
> > to
> > > > > point
> > > > > > 1,
> > > > > > > > > > perhaps
> > > > > > > > > > >> >we can introduce it to other window-related operators
> > in
> > > > the
> > > > > > > > future.>
> > > > > > > > > > 3.
> > > > > > > > > > >> >You need to modify the
> > FlinkChangelogModeInferenceProgram
> > > > to
> > > > > > > ensure
> > > > > > > > > > that
> > > > > > > > > > >> >downstream nodes of interval joins with early firing
> > > > enabled
> > > > > > are
> > > > > > > > > aware
> > > > > > > > > > of
> > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > >> >
> > > > > > > > > > >> > - We agree that time_mode could be introduced to
> > other
> > > > > > > > > window-based
> > > > > > > > > > >> > operators down the road. We also want to support
> > early
> > > > > fire
> > > > > > > for
> > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added the
> code
> > > > > change
> > > > > > on
> > > > > > > > it
> > > > > > > > > > >> > <
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > >> >
> > > > > > > > > > >> > in the proposal.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >Thanks again for your time and feedback! I’ve updated
> > the
> > > > > > > proposal
> > > > > > > > > with
> > > > > > > > > > >> >these points. Please let me know if there’s anything
> > > else I
> > > > > > > should
> > > > > > > > > > >> address.
> > > > > > > > > > >> >
> > > > > > > > > > >> >Best,
> > > > > > > > > > >> >Weiqing
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > xyzhong...@163.com>
> > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP. I
> > have a
> > > > few
> > > > > > > > > > questions:
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> 1. Currently, there are some configs related to
> early
> > > > > firing
> > > > > > > > > > available
> > > > > > > > > > >> to
> > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although their
> > > > > > > documentation
> > > > > > > > > > states
> > > > > > > > > > >> >> that they are only applicable to the Window
> operator,
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> it seems possible that they may also be relevant in
> > the
> > > > > > context
> > > > > > > > of
> > > > > > > > > > this
> > > > > > > > > > >> >> FLIP. Otherwise, having different early firing
> > > behaviors
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> 2. The design of `time_mode` is excellent. Similar
> to
> > > > point
> > > > > > 1,
> > > > > > > > > > perhaps
> > > > > > > > > > >> we
> > > > > > > > > > >> >> can introduce it to other window-related operators
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> in the future.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> 3. You need to modify the
> > > > > FlinkChangelogModeInferenceProgram
> > > > > > to
> > > > > > > > > > ensure
> > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> early firing enabled are aware of retract or upsert
> > > > > messages.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> --
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> Best！
> > > > > > > > > > >> >> Xuyang
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > xingc...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early fire
> for
> > > time
> > > > > > > > interval
> > > > > > > > > > outer
> > > > > > > > > > >> >> >joins is feasible overall. I have a few questions.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >1. The current output stream of a time interval
> > outer
> > > > join
> > > > > > is
> > > > > > > an
> > > > > > > > > > >> >> >append-only stream. This change will make it a
> > > potential
> > > > > > > > > retractable
> > > > > > > > > > >> >> >stream. I'm not sure if the planner supports a
> > dynamic
> > > > > > output
> > > > > > > > type
> > > > > > > > > > like
> > > > > > > > > > >> >> >that. Could you add this part to your design doc?
> > > > > > > > > > >> >> >2. What's the use case when the downstream
> > components
> > > > need
> > > > > > to
> > > > > > > > get
> > > > > > > > > > the
> > > > > > > > > > >> >> early
> > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > >> >> >3. The time interval join operator itself is not
> > quite
> > > > > > > efficient
> > > > > > > > > > when
> > > > > > > > > > >> the
> > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > overhead
> > > > > after
> > > > > > > > > > introducing
> > > > > > > > > > >> >> >this feature?
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > > > yangweiqing...@gmail.com>
> > > > > > > > > > >> >> >wrote:
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Just a gentle reminder regarding the proposal I
> > > shared
> > > > > on
> > > > > > > > early
> > > > > > > > > > fire
> > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> greatly
> > > > > > appreciate
> > > > > > > > > your
> > > > > > > > > > >> >> feedback
> > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Here’s the link to the proposal document: Link
> > > > > > > > > > >> >> >> <
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Best,
> > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <
> > > > > > > > > > >> yangweiqing...@gmail.com
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > introducing
> > > > > > early
> > > > > > > > fire
> > > > > > > > > > >> support
> > > > > > > > > > >> >> >> for
> > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > >> >> >> > In many streaming applications, particularly
> > > > real-time
> > > > > > > > > analytics
> > > > > > > > > > >> and
> > > > > > > > > > >> >> >> > monitoring systems, it is valuable to obtain
> > > partial
> > > > > > > results
> > > > > > > > > > >> earlier
> > > > > > > > > > >> >> >> rather
> > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > finalize.
> > > > For
> > > > > > > Flink
> > > > > > > > > SQL
> > > > > > > > > > >> >> interval
> > > > > > > > > > >> >> >> > joins, results are typically delayed until
> > > > watermarks
> > > > > > > ensure
> > > > > > > > > no
> > > > > > > > > > >> more
> > > > > > > > > > >> >> >> > matches can occur. This delay can be
> challenging
> > > for
> > > > > > > > scenarios
> > > > > > > > > > that
> > > > > > > > > > >> >> >> require
> > > > > > > > > > >> >> >> > fast feedback. Early fire support addresses
> this
> > > by
> > > > > > > emitting
> > > > > > > > > > >> >> intermediate
> > > > > > > > > > >> >> >> > results speculatively and using retractions or
> > > > updates
> > > > > > to
> > > > > > > > > > maintain
> > > > > > > > > > >> >> >> eventual
> > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > >> >> >> > <
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to refine
> > this
> > > > > > > feature.
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > >> >> >> >
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
