Thanks for updating the FLIP, Weiqing.
Another question, what is the granularity of the early fire hint? Will the
hint be applied to all the outer joins in the same block? What if in the
query block there are also aggregations? For example:
SELECT window_start, window_end, buyer, count(*) /*+ EARLY_FIRE('delay'='5s'
, 'time_mode'='rowtime|proctime') */
FROM Orders o LEFT JOIN Shipments s
WHERE o.id = s.order_id
AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time
GROUP BY o.window_start, o.window_end, buyer;
What will be the behavior? Will both the aggregation and interval join? Or
will it only be applied to the join?
It would be useful to define the behavior here.
Thanks,
Jiangjie (Becket) Qin
On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <yangweiqing...@gmail.com>
wrote:
> Thanks for the feedback, Becket. I’ve heard similar concerns from other
> reviewers, and I now agree that the `interval` option is likely not widely
> used. In the FLIP, it was intended specifically for Interval Join, not as a
> general early-fire mechanism. We can consider making it more generic in a
> future update. Since it’s optional, I’ll go ahead and remove it for
> clarity.
>
> Appreciate your input!
>
> Best,
> Weiqing
>
> On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <becket....@gmail.com> wrote:
>
> > Sorry for the late reply. I have a question regarding the configuration
> of
> > interval. I am wondering in which scenario the interval config will
> > actually be used. Is the interval configuration only useful for windowed
> > aggregation?
> >
> > For the join operations, early fire seems only applicable to the outer
> join
> > before there is a match. Once there is a match, the output of the join
> > operation should be just event driven. For example, consider the
> following
> > scenario:
> > 1. left outer join, and there is no match from the right side before the
> > initial delay
> > 2. a record [left, null] was emitted due to early fire
> > 3. one of the following two cases must happen
> > a. there is still no match event from the right side before the
> > specific interval passes. In this case, we are not supposed to emit
> another
> > [left, null].
> > b. if there is a match, a record of [left, right] should be emitted
> > immediately regardless of the interval. If this happens, the interval
> will
> > be ignored from this point on.
> >
> > That said, if the configuration proposed in this FLIP is intended not
> only
> > for join, but for general purpose early fire, then the interval config
> > makes sense.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <yangweiqing...@gmail.com>
> > wrote:
> >
> > > Thank you all for reviewing. As there are no objections, I’ll move
> > forward
> > > with a vote.
> > >
> > > Best regards,
> > > Weiqing
> > >
> > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> > > vsowr...@asu.edu>
> > > wrote:
> > >
> > > > Same here, I don't have any more questions. Thanks!
> > > >
> > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xingc...@gmail.com>
> wrote:
> > > >
> > > > > Hi Weiqing,
> > > > >
> > > > > I don't have any more questions. The doc looks good to me.
> > > > >
> > > > > Thanks,
> > > > > Xingcan
> > > > >
> > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > > > vsowr...@asu.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Weiqing,
> > > > > >
> > > > > > Thanks, that makes sense! Looks like I missed it.
> > > > > >
> > > > > > Regards
> > > > > > Venkata krishnan
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > > > yangweiqing...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Venkata,
> > > > > > >
> > > > > > > * > where only one earlyFire is fired The DELAY *
> > > > > > >
> > > > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > > > >
> > > > > > > of the proposal can produce a single early fire per interval,
> > > > aligning
> > > > > > with
> > > > > > > your suggestion. How it integrates with existing early-fire
> > > > > > configurations
> > > > > > > are mentioned here
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > > > >.
> > > > > > > Let me know if you have any further questions or feedback!
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Weiqing
> > > > > > >
> > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > > > vsowr...@asu.edu> wrote:
> > > > > > >
> > > > > > > > Thanks for the response!
> > > > > > > >
> > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > early-fire
> > > > > > > outputs
> > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > once.
> > > > > After
> > > > > > > that
> > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > >
> > > > > > > > One more question, would it make sense to support a mechanism
> > > > through
> > > > > > > > configuration where only one earlyFire is fired for an
> interval
> > > or
> > > > > > > window?
> > > > > > > > This would also simplify the state overhead for use cases
> where
> > > > only
> > > > > > one
> > > > > > > > early fire is required within a window or interval. Thoughts?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Venkata krishnan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > > > yangweiqing...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Venkata,
> > > > > > > > >
> > > > > > > > > Thanks for your feedback!
> > > > > > > > >
> > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > early-fire
> > > > > > > outputs
> > > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > > once.
> > > > > > After
> > > > > > > > that
> > > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > > >
> > > > > > > > > We haven’t included a late-fire mechanism for interval
> joins
> > in
> > > > > this
> > > > > > > > FLIP,
> > > > > > > > > but it’s certainly something we can consider for future
> > > efforts!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Weiqing
> > > > > > > > >
> > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan Sowrirajan
> <
> > > > > > > > > vsowr...@asu.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Weiqing,
> > > > > > > > > >
> > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also
> instead
> > > of
> > > > > > > > > controlling
> > > > > > > > > > and configuring through Flink configs unlike the older
> > window
> > > > > > > > > aggregation,
> > > > > > > > > > hints seems to be a much better approach. This enables a
> > > > > > customizable
> > > > > > > > > early
> > > > > > > > > > fire behavior for individual interval joins.
> > > > > > > > > >
> > > > > > > > > > Couple of questions:
> > > > > > > > > >
> > > > > > > > > > 1. Does the *early fire* emit an output every
> > > earlyFireInterval
> > > > > > time
> > > > > > > or
> > > > > > > > > > will it be a one time output emission and another output
> > > > emitted
> > > > > at
> > > > > > > the
> > > > > > > > > end
> > > > > > > > > > of the interval?
> > > > > > > > > > 2. Are there plans to support *late fire *similar to the
> > > window
> > > > > > > > > > aggregations in later FLIPs?
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Venkata krishnan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > > > yangweiqing...@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > > > >
> > > > > > > > > > > Xingcan (@xingc...@gmail.com) – do you have any
> > concerns?
> > > > > > > > > > >
> > > > > > > > > > > If no further objections arise from anyone, I’ll
> proceed
> > to
> > > > > mark
> > > > > > > FLIP
> > > > > > > > > as
> > > > > > > > > > > ready for voting.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Weiqing
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <
> > xyzhong...@163.com
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > LGTM overall. Thanks for updating. I have no problem
> > and
> > > +1
> > > > > for
> > > > > > > > this
> > > > > > > > > > > > feature.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Best！
> > > > > > > > > > > > Xuyang
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > > > yangweiqing...@gmail.com
> > > > > > >
> > > > > > > > 写道：
> > > > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > > > >
> > > > > > > > > > > > >Thank you for your detailed feedback! I’ve updated
> the
> > > > > > proposal
> > > > > > > > doc
> > > > > > > > > > > > >accordingly. Please feel free to take another look
> and
> > > let
> > > > > me
> > > > > > > know
> > > > > > > > > if
> > > > > > > > > > > you
> > > > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > > > >
> > > > > > > > > > > > >Best regards,
> > > > > > > > > > > > >Weiqing
> > > > > > > > > > > > >
> > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > > > xyzhong...@163.com>
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> After reading the new FLIP, I have no issues with
> > the
> > > > part
> > > > > > > > `public
> > > > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > > > IntervalJoin
> > > > > > > > node,
> > > > > > > > > > > IIUC:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind of
> > the
> > > > > > > > IntervalJoin
> > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely on
> > the
> > > > > > > > > requirements
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - When early firing is disabled, the UpdateKind of
> > the
> > > > > > > > > IntervalJoin
> > > > > > > > > > is
> > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> - Nevertheless, whether early firing is enabled or
> > > > > disabled,
> > > > > > > the
> > > > > > > > > > > > >> IntervalJoin should always require its input to
> keep
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with
> > NONE.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> --
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Best！
> > > > > > > > > > > > >> Xuyang
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > > > yangweiqing...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >Thanks so much for the feedback - it was very
> > > helpful!
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 1. The current output stream of a time
> interval
> > > > outer
> > > > > > join
> > > > > > > > is
> > > > > > > > > an
> > > > > > > > > > > > >> >append-only stream. This change will make it a
> > > > potential
> > > > > > > > > > retractable
> > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> > > dynamic
> > > > > > output
> > > > > > > > > type
> > > > > > > > > > > like
> > > > > > > > > > > > >> >that. Could you add this part to your design
> doc?*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - Yes, enabling early firing on time interval
> > > outer
> > > > > > joins
> > > > > > > > can
> > > > > > > > > > emit
> > > > > > > > > > > > >> > retractions when previously emitted rows are
> > > updated
> > > > > or
> > > > > > > > > > > invalidated
> > > > > > > > > > > > by
> > > > > > > > > > > > >> > later matches. I’ve updated the proposal
> > (Planner
> > > > > > > Awareness
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > and Changes in
> > FlinkChangelogModeInferenceProgram
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > >> >)
> > > > > > > > > > > > >> > to clarify that the stream might switch from
> > > > > append-only
> > > > > > > to
> > > > > > > > a
> > > > > > > > > > > > >> > retract/upsert stream. Let me know if anything
> > is
> > > > > > missing.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > > > components
> > > > > > need
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - The new INTERVAL option (in addition to
> DELAY)
> > > > > allows
> > > > > > > > > periodic
> > > > > > > > > > > > >> updates
> > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial
> > delay.
> > > > This
> > > > > > > > > captures
> > > > > > > > > > > how
> > > > > > > > > > > > >> results
> > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > > > “Repeatedly”
> > > > > > > > > option.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 3. The time interval join operator itself is
> not
> > > > quite
> > > > > > > > > efficient
> > > > > > > > > > > > when
> > > > > > > > > > > > >> >the state becomes large. Will there be any extra
> > > > overhead
> > > > > > > after
> > > > > > > > > > > > >> introducing
> > > > > > > > > > > > >> >this feature?*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > > > potentially
> > > > > > > > > > emitting
> > > > > > > > > > > > >> > partial matches multiple times with retraction
> > > > > (avoiding
> > > > > > > > > > duplicate
> > > > > > > > > > > > >> outputs
> > > > > > > > > > > > >> > though). However, if it’s disabled, there is
> no
> > > > > > additional
> > > > > > > > > cost.
> > > > > > > > > > > > Most
> > > > > > > > > > > > >> users
> > > > > > > > > > > > >> > find the performance trade-off acceptable for
> > the
> > > > > > > real-time
> > > > > > > > > > > > insights it
> > > > > > > > > > > > >> > provides.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 1. Currently, there are some configs related
> to
> > > > early
> > > > > > > firing
> > > > > > > > > > > > available
> > > > > > > > > > > > >> >to users: `table.exec.emit.early-fire.en**abled`
> > and
> > > > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > > > >> >**lay`.
> > > > > > > > > > > > >> >Although their documentation states that they are
> > > only
> > > > > > > > applicable
> > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >Window operator, it seems possible that they may
> > also
> > > > be
> > > > > > > > relevant
> > > > > > > > > > in
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >context of this FLIP. Otherwise, having different
> > > early
> > > > > > > firing
> > > > > > > > > > > > behaviors
> > > > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to avoid
> > > > > > confusion.
> > > > > > > > I’ve
> > > > > > > > > > > > added a
> > > > > > > > > > > > >> > section
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >in
> > > > > > > > > > > > >> > the proposal highlighting that we should align
> > > > > > hint-based
> > > > > > > > > > interval
> > > > > > > > > > > > join
> > > > > > > > > > > > >> > configurations with the existing
> > table.exec.emit.*
> > > > > > > settings.
> > > > > > > > > > > > >> Suggestions
> > > > > > > > > > > > >> > on how to make the unification are welcome! We
> > > plan
> > > > to
> > > > > > > > extend
> > > > > > > > > > > early
> > > > > > > > > > > > >> firing
> > > > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent.
> > Similar
> > > > to
> > > > > > > point
> > > > > > > > 1,
> > > > > > > > > > > > perhaps
> > > > > > > > > > > > >> >we can introduce it to other window-related
> > operators
> > > > in
> > > > > > the
> > > > > > > > > > future.>
> > > > > > > > > > > > 3.
> > > > > > > > > > > > >> >You need to modify the
> > > > FlinkChangelogModeInferenceProgram
> > > > > > to
> > > > > > > > > ensure
> > > > > > > > > > > > that
> > > > > > > > > > > > >> >downstream nodes of interval joins with early
> > firing
> > > > > > enabled
> > > > > > > > are
> > > > > > > > > > > aware
> > > > > > > > > > > > of
> > > > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > - We agree that time_mode could be introduced
> to
> > > > other
> > > > > > > > > > > window-based
> > > > > > > > > > > > >> > operators down the road. We also want to
> support
> > > > early
> > > > > > > fire
> > > > > > > > > for
> > > > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added
> the
> > > code
> > > > > > > change
> > > > > > > > on
> > > > > > > > > > it
> > > > > > > > > > > > >> > <
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > in the proposal.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve
> > updated
> > > > the
> > > > > > > > > proposal
> > > > > > > > > > > with
> > > > > > > > > > > > >> >these points. Please let me know if there’s
> > anything
> > > > > else I
> > > > > > > > > should
> > > > > > > > > > > > >> address.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >Best,
> > > > > > > > > > > > >> >Weiqing
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > > > xyzhong...@163.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this FLIP.
> I
> > > > have a
> > > > > > few
> > > > > > > > > > > > questions:
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> 1. Currently, there are some configs related to
> > > early
> > > > > > > firing
> > > > > > > > > > > > available
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled` and
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although
> > their
> > > > > > > > > documentation
> > > > > > > > > > > > states
> > > > > > > > > > > > >> >> that they are only applicable to the Window
> > > operator,
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> it seems possible that they may also be
> relevant
> > in
> > > > the
> > > > > > > > context
> > > > > > > > > > of
> > > > > > > > > > > > this
> > > > > > > > > > > > >> >> FLIP. Otherwise, having different early firing
> > > > > behaviors
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent.
> > Similar
> > > to
> > > > > > point
> > > > > > > > 1,
> > > > > > > > > > > > perhaps
> > > > > > > > > > > > >> we
> > > > > > > > > > > > >> >> can introduce it to other window-related
> > operators
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> in the future.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> 3. You need to modify the
> > > > > > > FlinkChangelogModeInferenceProgram
> > > > > > > > to
> > > > > > > > > > > > ensure
> > > > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> early firing enabled are aware of retract or
> > upsert
> > > > > > > messages.
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> --
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> Best！
> > > > > > > > > > > > >> >> Xuyang
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > > > xingc...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early
> fire
> > > for
> > > > > time
> > > > > > > > > > interval
> > > > > > > > > > > > outer
> > > > > > > > > > > > >> >> >joins is feasible overall. I have a few
> > questions.
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >1. The current output stream of a time
> interval
> > > > outer
> > > > > > join
> > > > > > > > is
> > > > > > > > > an
> > > > > > > > > > > > >> >> >append-only stream. This change will make it a
> > > > > potential
> > > > > > > > > > > retractable
> > > > > > > > > > > > >> >> >stream. I'm not sure if the planner supports a
> > > > dynamic
> > > > > > > > output
> > > > > > > > > > type
> > > > > > > > > > > > like
> > > > > > > > > > > > >> >> >that. Could you add this part to your design
> > doc?
> > > > > > > > > > > > >> >> >2. What's the use case when the downstream
> > > > components
> > > > > > need
> > > > > > > > to
> > > > > > > > > > get
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >> early
> > > > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > > > >> >> >3. The time interval join operator itself is
> not
> > > > quite
> > > > > > > > > efficient
> > > > > > > > > > > > when
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > > > overhead
> > > > > > > after
> > > > > > > > > > > > introducing
> > > > > > > > > > > > >> >> >this feature?
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <
> > > > > > > > > > > > yangweiqing...@gmail.com>
> > > > > > > > > > > > >> >> >wrote:
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the
> proposal
> > I
> > > > > shared
> > > > > > > on
> > > > > > > > > > early
> > > > > > > > > > > > fire
> > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> > > greatly
> > > > > > > > appreciate
> > > > > > > > > > > your
> > > > > > > > > > > > >> >> feedback
> > > > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Here’s the link to the proposal document:
> Link
> > > > > > > > > > > > >> >> >> <
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> Best,
> > > > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing
> Yang
> > <
> > > > > > > > > > > > >> yangweiqing...@gmail.com
> > > > > > > > > > > > >> >> >
> > > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > > > introducing
> > > > > > > > early
> > > > > > > > > > fire
> > > > > > > > > > > > >> support
> > > > > > > > > > > > >> >> >> for
> > > > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > > > >> >> >> > In many streaming applications,
> particularly
> > > > > > real-time
> > > > > > > > > > > analytics
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to
> obtain
> > > > > partial
> > > > > > > > > results
> > > > > > > > > > > > >> earlier
> > > > > > > > > > > > >> >> >> rather
> > > > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > > > finalize.
> > > > > > For
> > > > > > > > > Flink
> > > > > > > > > > > SQL
> > > > > > > > > > > > >> >> interval
> > > > > > > > > > > > >> >> >> > joins, results are typically delayed until
> > > > > > watermarks
> > > > > > > > > ensure
> > > > > > > > > > > no
> > > > > > > > > > > > >> more
> > > > > > > > > > > > >> >> >> > matches can occur. This delay can be
> > > challenging
> > > > > for
> > > > > > > > > > scenarios
> > > > > > > > > > > > that
> > > > > > > > > > > > >> >> >> require
> > > > > > > > > > > > >> >> >> > fast feedback. Early fire support
> addresses
> > > this
> > > > > by
> > > > > > > > > emitting
> > > > > > > > > > > > >> >> intermediate
> > > > > > > > > > > > >> >> >> > results speculatively and using
> retractions
> > or
> > > > > > updates
> > > > > > > > to
> > > > > > > > > > > > maintain
> > > > > > > > > > > > >> >> >> eventual
> > > > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > > > >> >> >> > <
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to
> > refine
> > > > this
> > > > > > > > > feature.
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > >> >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
