Hi, Weiqing. Thank you for drafting this FLIP. I have a few questions:
1. Currently, there are some configs related to early firing available to
users: `table.exec.emit.early-fire.enabled` and
`table.exec.emit.early-fire.delay`. Although their documentation states that
they are only applicable to the Window operator,
it seems possible that they may also be relevant in the context of this FLIP.
Otherwise, having different early firing behaviors
for different operators could confuse users.
2. The design of `time_mode` is excellent. Similar to point 1, perhaps we can
introduce it to other window-related operators
in the future.
3. You need to modify the FlinkChangelogModeInferenceProgram to ensure that
downstream nodes of interval joins with
early firing enabled are aware of retract or upsert messages.
--
Best！
Xuyang
At 2025-01-07 06:35:51, "Xingcan Cui" <xingc...@gmail.com> wrote:
>Hi Weiqing,
>
>Thanks for the proposal. IMO, adding early fire for time interval outer
>joins is feasible overall. I have a few questions.
>
>1. The current output stream of a time interval outer join is an
>append-only stream. This change will make it a potential retractable
>stream. I'm not sure if the planner supports a dynamic output type like
>that. Could you add this part to your design doc?
>2. What's the use case when the downstream components need to get the early
>fired results regularly?
>3. The time interval join operator itself is not quite efficient when the
>state becomes large. Will there be any extra overhead after introducing
>this feature?
>
>Thanks,
>Xingcan
>
>On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <yangweiqing...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> Just a gentle reminder regarding the proposal I shared on early fire
>> support for Flink SQL interval joins. I’d greatly appreciate your feedback
>> or suggestions.
>>
>> Here’s the link to the proposal document: Link
>> <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >
>>
>> Thank you!
>>
>> Best,
>> Weiqing
>>
>> On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <yangweiqing...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > I’d like to initiate a discussion about introducing early fire support
>> for
>> > Flink SQL interval joins.
>> >
>> > *Motivation*
>> > In many streaming applications, particularly real-time analytics and
>> > monitoring systems, it is valuable to obtain partial results earlier
>> rather
>> > than waiting for full join conditions to finalize. For Flink SQL interval
>> > joins, results are typically delayed until watermarks ensure no more
>> > matches can occur. This delay can be challenging for scenarios that
>> require
>> > fast feedback. Early fire support addresses this by emitting intermediate
>> > results speculatively and using retractions or updates to maintain
>> eventual
>> > consistency and ensure correctness.
>> >
>> > Here’s the proposal document: Link
>> > <
>> https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph
>> >
>> >
>> > Your feedback and ideas are welcome to refine this feature.
>> >
>> > Thanks,
>> > Weiqing
>> >
>>
