Hi Weiqing, Thanks for the proposal. IMO, adding early fire for time interval outer joins is feasible overall. I have a few questions.
1. The current output stream of a time interval outer join is an append-only stream. This change will make it a potential retractable stream. I'm not sure if the planner supports a dynamic output type like that. Could you add this part to your design doc? 2. What's the use case when the downstream components need to get the early fired results regularly? 3. The time interval join operator itself is not quite efficient when the state becomes large. Will there be any extra overhead after introducing this feature? Thanks, Xingcan On Mon, Jan 6, 2025 at 4:11 PM Weiqing Yang <yangweiqing...@gmail.com> wrote: > Hi all, > > Just a gentle reminder regarding the proposal I shared on early fire > support for Flink SQL interval joins. I’d greatly appreciate your feedback > or suggestions. > > Here’s the link to the proposal document: Link > < > https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph > > > > Thank you! > > Best, > Weiqing > > On Sun, Dec 22, 2024 at 11:19 PM Weiqing Yang <yangweiqing...@gmail.com> > wrote: > > > Hi all, > > > > I’d like to initiate a discussion about introducing early fire support > for > > Flink SQL interval joins. > > > > *Motivation* > > In many streaming applications, particularly real-time analytics and > > monitoring systems, it is valuable to obtain partial results earlier > rather > > than waiting for full join conditions to finalize. For Flink SQL interval > > joins, results are typically delayed until watermarks ensure no more > > matches can occur. This delay can be challenging for scenarios that > require > > fast feedback. Early fire support addresses this by emitting intermediate > > results speculatively and using retractions or updates to maintain > eventual > > consistency and ensure correctness. > > > > Here’s the proposal document: Link > > < > https://docs.google.com/document/d/1YobpNdnvzSsceniVj4NZWi445gb1-54Rox-D7nPArZo/edit?tab=t.0#heading=h.z7bl0h2hwkph > > > > > > Your feedback and ideas are welcome to refine this feature. > > > > Thanks, > > Weiqing > > >
