Hi Becket,
Thanks for the feedback! I’ve updated the design doc with a new
“Granularity of the Early Fire Hint” section: link
<https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.gle8q83bjlj5>
.
Here’s a quick summary:
- EARLY_FIRE is scoped to a single operator. If multiple operators in
one query can support early firing, Flink applies the hint to the first
matching operator and logs a warning to highlight the ambiguity. Subsequent
operators will not receive the hint.
- Split queries for different early-fire settings. If you need separate
early-fire configurations - or just want to avoid ambiguity - splitting the
SQL into multiple statements or views ensures each operator has its own
distinct EARLY_FIRE hint.
Please let me know if you have any other questions or suggestions.
Best,
Weiqing
On Fri, Feb 28, 2025 at 5:01 PM Becket Qin <becket....@gmail.com> wrote:
> Thanks for updating the FLIP, Weiqing.
>
> Another question, what is the granularity of the early fire hint? Will the
> hint be applied to all the outer joins in the same block? What if in the
> query block there are also aggregations? For example:
>
> SELECT window_start, window_end, buyer, count(*) /*+
> EARLY_FIRE('delay'='5s'
> , 'time_mode'='rowtime|proctime') */
> FROM Orders o LEFT JOIN Shipments s
> WHERE o.id = s.order_id
> AND o.order_time BETWEEN s.ship_time - INTERVAL '4' HOUR AND s.ship_time
> GROUP BY o.window_start, o.window_end, buyer;
>
> What will be the behavior? Will both the aggregation and interval join? Or
> will it only be applied to the join?
> It would be useful to define the behavior here.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Feb 27, 2025 at 2:31 PM Weiqing Yang <yangweiqing...@gmail.com>
> wrote:
>
> > Thanks for the feedback, Becket. I’ve heard similar concerns from other
> > reviewers, and I now agree that the `interval` option is likely not
> widely
> > used. In the FLIP, it was intended specifically for Interval Join, not
> as a
> > general early-fire mechanism. We can consider making it more generic in a
> > future update. Since it’s optional, I’ll go ahead and remove it for
> > clarity.
> >
> > Appreciate your input!
> >
> > Best,
> > Weiqing
> >
> > On Wed, Feb 19, 2025 at 9:46 AM Becket Qin <becket....@gmail.com> wrote:
> >
> > > Sorry for the late reply. I have a question regarding the configuration
> > of
> > > interval. I am wondering in which scenario the interval config will
> > > actually be used. Is the interval configuration only useful for
> windowed
> > > aggregation?
> > >
> > > For the join operations, early fire seems only applicable to the outer
> > join
> > > before there is a match. Once there is a match, the output of the join
> > > operation should be just event driven. For example, consider the
> > following
> > > scenario:
> > > 1. left outer join, and there is no match from the right side before
> the
> > > initial delay
> > > 2. a record [left, null] was emitted due to early fire
> > > 3. one of the following two cases must happen
> > > a. there is still no match event from the right side before the
> > > specific interval passes. In this case, we are not supposed to emit
> > another
> > > [left, null].
> > > b. if there is a match, a record of [left, right] should be emitted
> > > immediately regardless of the interval. If this happens, the interval
> > will
> > > be ignored from this point on.
> > >
> > > That said, if the configuration proposed in this FLIP is intended not
> > only
> > > for join, but for general purpose early fire, then the interval config
> > > makes sense.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Jan 27, 2025 at 1:56 PM Weiqing Yang <yangweiqing...@gmail.com
> >
> > > wrote:
> > >
> > > > Thank you all for reviewing. As there are no objections, I’ll move
> > > forward
> > > > with a vote.
> > > >
> > > > Best regards,
> > > > Weiqing
> > > >
> > > > On Mon, Jan 27, 2025 at 9:30 AM Venkatakrishnan Sowrirajan <
> > > > vsowr...@asu.edu>
> > > > wrote:
> > > >
> > > > > Same here, I don't have any more questions. Thanks!
> > > > >
> > > > > On Sat, Jan 25, 2025, 10:10 PM Xingcan Cui <xingc...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi Weiqing,
> > > > > >
> > > > > > I don't have any more questions. The doc looks good to me.
> > > > > >
> > > > > > Thanks,
> > > > > > Xingcan
> > > > > >
> > > > > > On Wed, Jan 22, 2025 at 8:46 PM Venkatakrishnan Sowrirajan <
> > > > > > vsowr...@asu.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Weiqing,
> > > > > > >
> > > > > > > Thanks, that makes sense! Looks like I missed it.
> > > > > > >
> > > > > > > Regards
> > > > > > > Venkata krishnan
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 21, 2025 at 10:55 PM Weiqing Yang <
> > > > > yangweiqing...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Venkata,
> > > > > > > >
> > > > > > > > * > where only one earlyFire is fired The DELAY *
> > > > > > > >
> > > > > > > > The DELAY option mentioned in the Public Interfaces section
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.yp9ng89zwc1z__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtXqs8qQL$
> > > > > > > > >
> > > > > > > > of the proposal can produce a single early fire per interval,
> > > > > aligning
> > > > > > > with
> > > > > > > > your suggestion. How it integrates with existing early-fire
> > > > > > > configurations
> > > > > > > > are mentioned here
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!cSzLIiXfVEVa9ClafsAoHYeOAxXCu5Em4XlW-MWWjtCsDAVCONBJEwxQ6kFXaCHxOtpVR7w6siu_q7x6HZbUtcJ04z2O$
> > > > > > > > >.
> > > > > > > > Let me know if you have any further questions or feedback!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Weiqing
> > > > > > > >
> > > > > > > > On Tue, Jan 21, 2025 at 11:30 AM Venkatakrishnan Sowrirajan <
> > > > > > > > vsowr...@asu.edu> wrote:
> > > > > > > >
> > > > > > > > > Thanks for the response!
> > > > > > > > >
> > > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > > early-fire
> > > > > > > > outputs
> > > > > > > > > will occur repeatedly at every earlyFireInterval, not just
> > > once.
> > > > > > After
> > > > > > > > that
> > > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > > >
> > > > > > > > > One more question, would it make sense to support a
> mechanism
> > > > > through
> > > > > > > > > configuration where only one earlyFire is fired for an
> > interval
> > > > or
> > > > > > > > window?
> > > > > > > > > This would also simplify the state overhead for use cases
> > where
> > > > > only
> > > > > > > one
> > > > > > > > > early fire is required within a window or interval.
> Thoughts?
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Venkata krishnan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 20, 2025 at 9:11 PM Weiqing Yang <
> > > > > > yangweiqing...@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Venkata,
> > > > > > > > > >
> > > > > > > > > > Thanks for your feedback!
> > > > > > > > > >
> > > > > > > > > > If the optional `interval` in the proposal is enabled,
> > > > early-fire
> > > > > > > > outputs
> > > > > > > > > > will occur repeatedly at every earlyFireInterval, not
> just
> > > > once.
> > > > > > > After
> > > > > > > > > that
> > > > > > > > > > interval concludes, there will also be a final emission.
> > > > > > > > > >
> > > > > > > > > > We haven’t included a late-fire mechanism for interval
> > joins
> > > in
> > > > > > this
> > > > > > > > > FLIP,
> > > > > > > > > > but it’s certainly something we can consider for future
> > > > efforts!
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Weiqing
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 16, 2025 at 3:27 PM Venkatakrishnan
> Sowrirajan
> > <
> > > > > > > > > > vsowr...@asu.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Weiqing,
> > > > > > > > > > >
> > > > > > > > > > > Thanks, this is a great addition to Flink SQL. Also
> > instead
> > > > of
> > > > > > > > > > controlling
> > > > > > > > > > > and configuring through Flink configs unlike the older
> > > window
> > > > > > > > > > aggregation,
> > > > > > > > > > > hints seems to be a much better approach. This enables
> a
> > > > > > > customizable
> > > > > > > > > > early
> > > > > > > > > > > fire behavior for individual interval joins.
> > > > > > > > > > >
> > > > > > > > > > > Couple of questions:
> > > > > > > > > > >
> > > > > > > > > > > 1. Does the *early fire* emit an output every
> > > > earlyFireInterval
> > > > > > > time
> > > > > > > > or
> > > > > > > > > > > will it be a one time output emission and another
> output
> > > > > emitted
> > > > > > at
> > > > > > > > the
> > > > > > > > > > end
> > > > > > > > > > > of the interval?
> > > > > > > > > > > 2. Are there plans to support *late fire *similar to
> the
> > > > window
> > > > > > > > > > > aggregations in later FLIPs?
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Venkata krishnan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jan 15, 2025 at 6:16 PM Weiqing Yang <
> > > > > > > > yangweiqing...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for reviewing, Xuyang!
> > > > > > > > > > > >
> > > > > > > > > > > > Xingcan (@xingc...@gmail.com) – do you have any
> > > concerns?
> > > > > > > > > > > >
> > > > > > > > > > > > If no further objections arise from anyone, I’ll
> > proceed
> > > to
> > > > > > mark
> > > > > > > > FLIP
> > > > > > > > > > as
> > > > > > > > > > > > ready for voting.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Weiqing
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 14, 2025 at 9:06 PM Xuyang <
> > > xyzhong...@163.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > LGTM overall. Thanks for updating. I have no
> problem
> > > and
> > > > +1
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > > > feature.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best！
> > > > > > > > > > > > > Xuyang
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 在 2025-01-15 12:33:16，"Weiqing Yang" <
> > > > > > yangweiqing...@gmail.com
> > > > > > > >
> > > > > > > > > 写道：
> > > > > > > > > > > > > >Hi Xuyang,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >Thank you for your detailed feedback! I’ve updated
> > the
> > > > > > > proposal
> > > > > > > > > doc
> > > > > > > > > > > > > >accordingly. Please feel free to take another look
> > and
> > > > let
> > > > > > me
> > > > > > > > know
> > > > > > > > > > if
> > > > > > > > > > > > you
> > > > > > > > > > > > > >have any further thoughts or suggestions.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >Best regards,
> > > > > > > > > > > > > >Weiqing
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >On Mon, Jan 13, 2025 at 3:50 AM Xuyang <
> > > > > xyzhong...@163.com>
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Hi, Weiqing.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> After reading the new FLIP, I have no issues
> with
> > > the
> > > > > part
> > > > > > > > > `public
> > > > > > > > > > > > > >> interface`. I only have some questions regarding
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> the details in the Proposed Changes section.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Regarding the ModifyKind and UpdateKind of the
> > > > > > IntervalJoin
> > > > > > > > > node,
> > > > > > > > > > > > IIUC:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - When early firing is enabled, the UpdateKind
> of
> > > the
> > > > > > > > > IntervalJoin
> > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > >> either ONLY_UPDATE_AFTER or
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> degrade to BEFORE_AND_AFTER, depending entirely
> on
> > > the
> > > > > > > > > > requirements
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> sink. And the ModifyKind is always ALL.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - When early firing is disabled, the UpdateKind
> of
> > > the
> > > > > > > > > > IntervalJoin
> > > > > > > > > > > is
> > > > > > > > > > > > > >> NONE, and the ModifyKind is INSERT.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> - Nevertheless, whether early firing is enabled
> or
> > > > > > disabled,
> > > > > > > > the
> > > > > > > > > > > > > >> IntervalJoin should always require its input to
> > keep
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> ModifyKind with INSERT_ONLY and UpdateKind with
> > > NONE.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> --
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Best！
> > > > > > > > > > > > > >> Xuyang
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> At 2025-01-09 15:30:44, "Weiqing Yang" <
> > > > > > > > > yangweiqing...@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >Hi Xingcan and Xuyang,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >Thanks so much for the feedback - it was very
> > > > helpful!
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 1. The current output stream of a time
> > interval
> > > > > outer
> > > > > > > join
> > > > > > > > > is
> > > > > > > > > > an
> > > > > > > > > > > > > >> >append-only stream. This change will make it a
> > > > > potential
> > > > > > > > > > > retractable
> > > > > > > > > > > > > >> >stream. I'm not sure if the planner supports a
> > > > dynamic
> > > > > > > output
> > > > > > > > > > type
> > > > > > > > > > > > like
> > > > > > > > > > > > > >> >that. Could you add this part to your design
> > doc?*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - Yes, enabling early firing on time
> interval
> > > > outer
> > > > > > > joins
> > > > > > > > > can
> > > > > > > > > > > emit
> > > > > > > > > > > > > >> > retractions when previously emitted rows are
> > > > updated
> > > > > > or
> > > > > > > > > > > > invalidated
> > > > > > > > > > > > > by
> > > > > > > > > > > > > >> > later matches. I’ve updated the proposal
> > > (Planner
> > > > > > > > Awareness
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.y5w17oloacws__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-WAM59Os$
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > and Changes in
> > > FlinkChangelogModeInferenceProgram
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > > >> >)
> > > > > > > > > > > > > >> > to clarify that the stream might switch from
> > > > > > append-only
> > > > > > > > to
> > > > > > > > > a
> > > > > > > > > > > > > >> > retract/upsert stream. Let me know if
> anything
> > > is
> > > > > > > missing.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 2. What's the use case when the downstream
> > > > > components
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >early fired results regularly?*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - The new INTERVAL option (in addition to
> > DELAY)
> > > > > > allows
> > > > > > > > > > periodic
> > > > > > > > > > > > > >> updates
> > > > > > > > > > > > > >> > (e.g., every 10 minutes) after the initial
> > > delay.
> > > > > This
> > > > > > > > > > captures
> > > > > > > > > > > > how
> > > > > > > > > > > > > >> results
> > > > > > > > > > > > > >> > evolve over time, similar to Apache Beam’s
> > > > > > “Repeatedly”
> > > > > > > > > > option.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 3. The time interval join operator itself is
> > not
> > > > > quite
> > > > > > > > > > efficient
> > > > > > > > > > > > > when
> > > > > > > > > > > > > >> >the state becomes large. Will there be any
> extra
> > > > > overhead
> > > > > > > > after
> > > > > > > > > > > > > >> introducing
> > > > > > > > > > > > > >> >this feature?*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - Early fire does introduce some overhead by
> > > > > > potentially
> > > > > > > > > > > emitting
> > > > > > > > > > > > > >> > partial matches multiple times with
> retraction
> > > > > > (avoiding
> > > > > > > > > > > duplicate
> > > > > > > > > > > > > >> outputs
> > > > > > > > > > > > > >> > though). However, if it’s disabled, there is
> > no
> > > > > > > additional
> > > > > > > > > > cost.
> > > > > > > > > > > > > Most
> > > > > > > > > > > > > >> users
> > > > > > > > > > > > > >> > find the performance trade-off acceptable
> for
> > > the
> > > > > > > > real-time
> > > > > > > > > > > > > insights it
> > > > > > > > > > > > > >> > provides.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 1. Currently, there are some configs related
> > to
> > > > > early
> > > > > > > > firing
> > > > > > > > > > > > > available
> > > > > > > > > > > > > >> >to users:
> `table.exec.emit.early-fire.en**abled`
> > > and
> > > > > > > > > > > > > >> >`table.exec.emit.early-fire.de <
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__http://table.exec.emit.early-fire.de__;!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-dmB0JB7$
> > > > > > > > > > > > > >> >**lay`.
> > > > > > > > > > > > > >> >Although their documentation states that they
> are
> > > > only
> > > > > > > > > applicable
> > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >Window operator, it seems possible that they
> may
> > > also
> > > > > be
> > > > > > > > > relevant
> > > > > > > > > > > in
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >context of this FLIP. Otherwise, having
> different
> > > > early
> > > > > > > > firing
> > > > > > > > > > > > > behaviors
> > > > > > > > > > > > > >> >for different operators could confuse users.*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - +1 on unifying early-fire behaviors to
> avoid
> > > > > > > confusion.
> > > > > > > > > I’ve
> > > > > > > > > > > > > added a
> > > > > > > > > > > > > >> > section
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.rr0i3gmdjt4q__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-cs7f7P2$
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >in
> > > > > > > > > > > > > >> > the proposal highlighting that we should
> align
> > > > > > > hint-based
> > > > > > > > > > > interval
> > > > > > > > > > > > > join
> > > > > > > > > > > > > >> > configurations with the existing
> > > table.exec.emit.*
> > > > > > > > settings.
> > > > > > > > > > > > > >> Suggestions
> > > > > > > > > > > > > >> > on how to make the unification are welcome!
> We
> > > > plan
> > > > > to
> > > > > > > > > extend
> > > > > > > > > > > > early
> > > > > > > > > > > > > >> firing
> > > > > > > > > > > > > >> > to window joins via hints in a future FLIP.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >*> 2. The design of `time_mode` is excellent.
> > > Similar
> > > > > to
> > > > > > > > point
> > > > > > > > > 1,
> > > > > > > > > > > > > perhaps
> > > > > > > > > > > > > >> >we can introduce it to other window-related
> > > operators
> > > > > in
> > > > > > > the
> > > > > > > > > > > future.>
> > > > > > > > > > > > > 3.
> > > > > > > > > > > > > >> >You need to modify the
> > > > > FlinkChangelogModeInferenceProgram
> > > > > > > to
> > > > > > > > > > ensure
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> >downstream nodes of interval joins with early
> > > firing
> > > > > > > enabled
> > > > > > > > > are
> > > > > > > > > > > > aware
> > > > > > > > > > > > > of
> > > > > > > > > > > > > >> >retract or upsert messages.*
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > - We agree that time_mode could be
> introduced
> > to
> > > > > other
> > > > > > > > > > > > window-based
> > > > > > > > > > > > > >> > operators down the road. We also want to
> > support
> > > > > early
> > > > > > > > fire
> > > > > > > > > > for
> > > > > > > > > > > > > >> > window join. Also, thanks for highlighting
> > > > > > > > > > > > > >> > FlinkChangelogModeInferenceProgram! I added
> > the
> > > > code
> > > > > > > > change
> > > > > > > > > on
> > > > > > > > > > > it
> > > > > > > > > > > > > >> > <
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z6qdwrvtgn4u__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-Y5SiJXB$
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > in the proposal.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >Thanks again for your time and feedback! I’ve
> > > updated
> > > > > the
> > > > > > > > > > proposal
> > > > > > > > > > > > with
> > > > > > > > > > > > > >> >these points. Please let me know if there’s
> > > anything
> > > > > > else I
> > > > > > > > > > should
> > > > > > > > > > > > > >> address.
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >Best,
> > > > > > > > > > > > > >> >Weiqing
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >On Mon, Jan 6, 2025 at 6:32 PM Xuyang <
> > > > > > xyzhong...@163.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >> Hi, Weiqing. Thank you for drafting this
> FLIP.
> > I
> > > > > have a
> > > > > > > few
> > > > > > > > > > > > > questions:
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> 1. Currently, there are some configs related
> to
> > > > early
> > > > > > > > firing
> > > > > > > > > > > > > available
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >> >> users: `table.exec.emit.early-fire.enabled`
> and
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> `table.exec.emit.early-fire.delay`. Although
> > > their
> > > > > > > > > > documentation
> > > > > > > > > > > > > states
> > > > > > > > > > > > > >> >> that they are only applicable to the Window
> > > > operator,
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> it seems possible that they may also be
> > relevant
> > > in
> > > > > the
> > > > > > > > > context
> > > > > > > > > > > of
> > > > > > > > > > > > > this
> > > > > > > > > > > > > >> >> FLIP. Otherwise, having different early
> firing
> > > > > > behaviors
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> for different operators could confuse users.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> 2. The design of `time_mode` is excellent.
> > > Similar
> > > > to
> > > > > > > point
> > > > > > > > > 1,
> > > > > > > > > > > > > perhaps
> > > > > > > > > > > > > >> we
> > > > > > > > > > > > > >> >> can introduce it to other window-related
> > > operators
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> in the future.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> 3. You need to modify the
> > > > > > > > FlinkChangelogModeInferenceProgram
> > > > > > > > > to
> > > > > > > > > > > > > ensure
> > > > > > > > > > > > > >> >> that downstream nodes of interval joins with
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> early firing enabled are aware of retract or
> > > upsert
> > > > > > > > messages.
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> --
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> Best！
> > > > > > > > > > > > > >> >> Xuyang
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >> >> At 2025-01-07 06:35:51, "Xingcan Cui" <
> > > > > > > xingc...@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> >> >Hi Weiqing,
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >Thanks for the proposal. IMO, adding early
> > fire
> > > > for
> > > > > > time
> > > > > > > > > > > interval
> > > > > > > > > > > > > outer
> > > > > > > > > > > > > >> >> >joins is feasible overall. I have a few
> > > questions.
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >1. The current output stream of a time
> > interval
> > > > > outer
> > > > > > > join
> > > > > > > > > is
> > > > > > > > > > an
> > > > > > > > > > > > > >> >> >append-only stream. This change will make
> it a
> > > > > > potential
> > > > > > > > > > > > retractable
> > > > > > > > > > > > > >> >> >stream. I'm not sure if the planner
> supports a
> > > > > dynamic
> > > > > > > > > output
> > > > > > > > > > > type
> > > > > > > > > > > > > like
> > > > > > > > > > > > > >> >> >that. Could you add this part to your design
> > > doc?
> > > > > > > > > > > > > >> >> >2. What's the use case when the downstream
> > > > > components
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > get
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> >> early
> > > > > > > > > > > > > >> >> >fired results regularly?
> > > > > > > > > > > > > >> >> >3. The time interval join operator itself is
> > not
> > > > > quite
> > > > > > > > > > efficient
> > > > > > > > > > > > > when
> > > > > > > > > > > > > >> the
> > > > > > > > > > > > > >> >> >state becomes large. Will there be any extra
> > > > > overhead
> > > > > > > > after
> > > > > > > > > > > > > introducing
> > > > > > > > > > > > > >> >> >this feature?
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >Thanks,
> > > > > > > > > > > > > >> >> >Xingcan
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang
> <
> > > > > > > > > > > > > yangweiqing...@gmail.com>
> > > > > > > > > > > > > >> >> >wrote:
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >> Hi all,
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Just a gentle reminder regarding the
> > proposal
> > > I
> > > > > > shared
> > > > > > > > on
> > > > > > > > > > > early
> > > > > > > > > > > > > fire
> > > > > > > > > > > > > >> >> >> support for Flink SQL interval joins. I’d
> > > > greatly
> > > > > > > > > appreciate
> > > > > > > > > > > > your
> > > > > > > > > > > > > >> >> feedback
> > > > > > > > > > > > > >> >> >> or suggestions.
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Here’s the link to the proposal document:
> > Link
> > > > > > > > > > > > > >> >> >> <
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Thank you!
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> Best,
> > > > > > > > > > > > > >> >> >> Weiqing
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> On Sun, Dec 22, 2024 at 11:19 PM Weiqing
> > Yang
> > > <
> > > > > > > > > > > > > >> yangweiqing...@gmail.com
> > > > > > > > > > > > > >> >> >
> > > > > > > > > > > > > >> >> >> wrote:
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >> >> > Hi all,
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > I’d like to initiate a discussion about
> > > > > > introducing
> > > > > > > > > early
> > > > > > > > > > > fire
> > > > > > > > > > > > > >> support
> > > > > > > > > > > > > >> >> >> for
> > > > > > > > > > > > > >> >> >> > Flink SQL interval joins.
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > *Motivation*
> > > > > > > > > > > > > >> >> >> > In many streaming applications,
> > particularly
> > > > > > > real-time
> > > > > > > > > > > > analytics
> > > > > > > > > > > > > >> and
> > > > > > > > > > > > > >> >> >> > monitoring systems, it is valuable to
> > obtain
> > > > > > partial
> > > > > > > > > > results
> > > > > > > > > > > > > >> earlier
> > > > > > > > > > > > > >> >> >> rather
> > > > > > > > > > > > > >> >> >> > than waiting for full join conditions to
> > > > > finalize.
> > > > > > > For
> > > > > > > > > > Flink
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > >> >> interval
> > > > > > > > > > > > > >> >> >> > joins, results are typically delayed
> until
> > > > > > > watermarks
> > > > > > > > > > ensure
> > > > > > > > > > > > no
> > > > > > > > > > > > > >> more
> > > > > > > > > > > > > >> >> >> > matches can occur. This delay can be
> > > > challenging
> > > > > > for
> > > > > > > > > > > scenarios
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> >> >> require
> > > > > > > > > > > > > >> >> >> > fast feedback. Early fire support
> > addresses
> > > > this
> > > > > > by
> > > > > > > > > > emitting
> > > > > > > > > > > > > >> >> intermediate
> > > > > > > > > > > > > >> >> >> > results speculatively and using
> > retractions
> > > or
> > > > > > > updates
> > > > > > > > > to
> > > > > > > > > > > > > maintain
> > > > > > > > > > > > > >> >> >> eventual
> > > > > > > > > > > > > >> >> >> > consistency and ensure correctness.
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > Here’s the proposal document: Link
> > > > > > > > > > > > > >> >> >> > <
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0*heading=h.z7bl0h2hwkph__;Iw!!IKRxdwAv5BmarQ!amkTjCPG108LnMxlN_eVP1GHgJpGNcvNJWSNr3NMfIoj0hTe4LvEKnFk0_gDXV0W-hozAXm9Kxw9VrlRT3jQ-ZfECmzD$
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > Your feedback and ideas are welcome to
> > > refine
> > > > > this
> > > > > > > > > > feature.
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >> > Thanks,
> > > > > > > > > > > > > >> >> >> > Weiqing
> > > > > > > > > > > > > >> >> >> >
> > > > > > > > > > > > > >> >> >>
> > > > > > > > > > > > > >> >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
