Hi Fabian, >Do you think this should be part of the proposal or would you be fine to leave this as future work?
Thanks for your response. As long as the potential for this extension is reflected in FLIP, leaving it as future work is ok for me. Best, Hongshun On Thu, Jun 4, 2026 at 1:24 AM Fabian Hueske <[email protected]> wrote: > Thanks Leonard, > > I've added a section about operator metrics to the proposal [1]. > If you have ideas for other useful metrics, please let me know and I'll add > them. > > Best, Fabian > > [1] > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421958523#FLIP579:LATERALSNAPSHOTJoin-Metrics > > Am Mi., 3. Juni 2026 um 10:18 Uhr schrieb Fabian Hueske < > [email protected] > >: > > > Thank you Hongshun for your feedback! > > > > You are right that restricting the probe-side input to append-only > changes > > deviates from the existing event-time (and disabled proc-time) temporal > > table joins. > > I chose this restriction because processing time semantics do not work > > well with retractions. > > A probe record of +(k1, p1) could join against a build-side of (k1, v1), > > while a later arriving probe-side retraction -(k1, p1) would join against > > (k1, v2). The resulting rectraction record of -(k1, p1, v2) would not > match > > the earlier insertion +(k1, p1, v1). > > The only changelog format that would work well would be upsert changes > > (+I, +UA, -D) with key-only deletes (under the assumption that the upsert > > key is still unique after the join). > > > > I would like to keep the default behavior of only accepting append-only > > inputs to prevent retraction mismatches. > > At the same time, the operator implementation only needs to be slightly > > adjusted to support arbitrary probe-side changes (if at all). The real > > change would be in the planner / rules. > > So allowing probe-side retraction input via a config option for > > power-users who know what they are doing is certainly possible. > > > > Do you think this should be part of the proposal or would you be fine to > > leave this as future work? > > > > Best, Fabian > > > > Am Mi., 3. Juni 2026 um 08:41 Uhr schrieb Leonard Xu <[email protected] > >: > > > >> Hi Fabian, > >> > >> Thanks for the detailed and thoughtful reply, and especially for > agreeing > >> to add operator metrics — that alone will significantly improve the > >> debuggability of the join in production. > >> > >> (1) On CPU spike / probe-side buffering > >> > >> +1 for your reasoning on both points — deferring InputSelectable until > >> the unaligned-checkpoint limitation is resolved, and leaving micro-batch > >> transition out of v1. Watermark alignment + probe-side scan-offset is > the > >> principled clean approach, agreed. > >> > >> (2) On Jark's backlog idea > >> > >> Between the two paths in your reply, I'd lean toward option (a) — having > >> the source connector emit a special WM (or a dedicated record > attribute) at > >> the end of backlog — as the long-term direction for exact flip-point > >> semantics. It naturally fits what is almost certainly going to be the > >> dominant production scenario: CDC sources (mysql-cdc, mongodb-cdc, > >> postgres-cdc, ...) as the dimension build-side, all of which already > have > >> an explicit "snapshot finished → binlog start" boundary internally. > >> > >> My suggestion is to ship FLIP-579 as proposed and collect user feedback > >> on LATERAL SNAPSHOT Join from real production usage — that will give us > >> solid evidence to prioritize this follow-up direction. > >> > >> (3) On the vote > >> > >> Once the metrics section is added and there are no further objections > >> from others, I'm fine to start the vote. > >> > >> > >> Best, > >> Leonard > >> > >> > >> > >> > 2026 6月 3 01:36，Fabian Hueske <[email protected]> 写道： > >> > > >> > Thanks for your valuable feedback Jark and Leonard! > >> > > >> > You are bringing up three of the tricky challenges that the new join > >> needs > >> > to deal with. > >> > > >> > (1) Jark: The build-side flip point is not exact > >> > > >> > This is correct. However, I would argue that a processing-time join > does > >> > not have exact guarantees anyway and can only produce roughly > >> time-aligned > >> > results. WM alignment of build and probe-side input should help to > keep > >> the > >> > alignment somewhat close. Of course, this does not mean that we > >> shouldn't > >> > try to give as good guarantees as possible. > >> > > >> > The primary mechanism for flipping from LOAD to JOIN phase is the > >> > build-side watermark crossing a configured point in time. Watermarks > are > >> > used to track progress and completeness in Flink. Using them as a > >> condition > >> > to switch from LOAD to JOIN phase, means that the build-side received > at > >> > least all changes up to that point in time. There might have been > >> changes > >> > with later timestamps as well. These could be buffered on the side to > >> have > >> > a stricter FLIP point, but IMO this additional data should be > tolerable > >> > under proc-time semantics. > >> > > >> > If the build-side input becomes stale, the processing idle timeout > flip > >> > condition gets applied. The assumption is that the build-side source > is > >> > currently exhausted and all data was consumed but the WM didn't > progress > >> > far enough to exceed the flip point. In this case, we want to flip and > >> > start the regular JOIN phase. > >> > > >> > For the use case of sources with an exact flip point, users would need > >> to > >> > know the timestamp of the last backlog record (or compute it if they > >> know > >> > roughly how long it takes to scan the backlog if it is computed > >> on-demand). > >> > I agree, this is not very practical. > >> > I can think of two options > >> > a) the source connector emits a special WM when it reaches the end of > >> the > >> > backlog. This would not require changes to the join operator but to > the > >> > source connectors. > >> > b) The design of the SNAPSHOT function has the > >> `load_completed_condition` > >> > which is an extension point to add logic to determine the flip point. > >> > > >> > > >> > (2) Jark: Buffering probe-side during LOAD phase > >> > > >> > I think this is very similar to Leonard's point about "LOAD phase > >> > backpressure" and also closely related to Leonard's point about > >> "Flip-point > >> > CPU spike". > >> > > >> > This is indeed a potential problem. If used without care, the > probe-side > >> > state might grow very large. Before talking about possible ways to > >> address > >> > this problem, let me explain how I think that the join would be used. > >> > > >> > A very common (maybe the most common?) use case should be to > initialize > >> the > >> > build-side input up to time t_b and then start processing the > probe-side > >> > input from time t_p (with t_p = t_b, or slightly less than t_b) > >> onwards. WM > >> > alignment would help to roughly align build-side and probe-side inputs > >> > (although not being perfectly aligned like the event-time join) > >> > Initializing the build-side up to t_b and starting consuming the > >> probe-side > >> > from t_p with (t_p << t_b) would mean that the first probe-side > records > >> are > >> > joined with much later versions of the build-side. > >> > > >> > The first scenario (t_p = t_b) can be controlled with WM alignment and > >> > scan-offset table hints on the probe-side input. Since the WMs of the > >> > build-side input would be less than the WMs of the probe-side input, > the > >> > probe-side input should be throttled until the build-side caught up > >> (which > >> > should be close the the flip point). > >> > Other scenarios (including the t_p << t_b scenario) would benefit from > >> an > >> > idea [1] that is described in the future work section of the FLIP. > That > >> > mechanism would also be based on WM alignment and would need some > >> > collaboration from the build-side source operator to indicate > >> completeness > >> > of the backlog. > >> > > >> > Using the InputSelectable interface is an idea that we also looked > into > >> (as > >> > Gustavo already pointed out). Unfortunately, it is incompatible with > >> > unaligned checkpoints and there are no other streaming operators that > >> > implement the interface. I haven't looked in depth at the current > >> > limitations, but if some of these would be resolved, it might be > >> possible > >> > to later extend the join operator. It might even work with relaxed > >> > guarantees because we don't need to fully block the input but just > >> throttle > >> > it such that less probe-side data needs to be buffered. > >> > > >> > The idea of limiting the size of the probe-side buffer with a config > >> like > >> > `max-buffer-size` sounds interesting. However, I'm not sure if > applying > >> > backpressure would really work because we still need to consume the > >> > build-side to be able to reach the flip point and selective > >> backpressure is > >> > not possible without InputSelectible. > >> > > >> > An earlier draft of the proposal described eager joining (now moved to > >> the > >> > future work section [2]). The idea is that a probe-side record would > be > >> > directly joined when a match was present or received during the LOAD > >> phase. > >> > After joining it wouldn't be put into state. This would of course > >> > significantly reduce the state size and solved the issue of CPU spikes > >> > during the transition but come at the cost of hard-to-explain > semantics > >> > (the current semantics are rather simple, we collect until the flip > >> point > >> > and join against that). > >> > Also, the idea of eager joining was developed when the join operator > was > >> > still restricted to FK-PK joins (single build-side record per join > key). > >> > The design generalized this restriction to arbitrary joins which means > >> > there might be more build-side matches for a probe-side record such > that > >> > the presence of a single join match (of a possibly earlier version) > does > >> > not guarantee completeness anymore. That's why the idea of eager > joining > >> > was discarded for now. > >> > > >> > > >> > (3) Leonard: Flip-point CPU spikes > >> > > >> > This is also a very valid concern. I would argue that the primary > >> mechanism > >> > to address this point should be to reduce the amount of buffered > >> probe-side > >> > records (see point (2) above). > >> > > >> > I also thought about your idea to micro-batch the draining. In the > >> design, > >> > the transition join is triggered per-key by event-time timers that > also > >> > emit the "current" probe-side WM downstream. If we want to continue > >> using > >> > this mechanism, we would need to schedule multiple timers and probably > >> use > >> > some clever mechanism to gradually advance probe-side WMs while still > >> > consuming records. There could be a separate "TRANSITION" phase during > >> > which we still append to the probe-buffer but use the mechanism for > >> atomic > >> > build-side updates. However, this would significantly complicate the > >> design > >> > affecting the control flow and recovery. Hence, I would first try to > >> > address this issue by reducing the probe-side state. > >> > > >> > If you have better ideas for how to use micro-batching during flip > >> > transition, I'm very open to exploring those. > >> > > >> > Leonard also brought up a very important point about metrics! > >> > I will add a section on operator metrics that will help to understand > >> the > >> > state of the operator. > >> > > >> > Please let me know your thoughts! > >> > > >> > Best, Fabian > >> > > >> > [1] > >> > > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421958523#FLIP579:LATERALSNAPSHOTJoin-ReduceBufferingofProbe-SideviaBuild-SideWatermarkSuppression > >> > [2] > >> > > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=421958523#FLIP579:LATERALSNAPSHOTJoin-EagerjoinmodeduringLOADphase > >> > > >> > > >> > Am Mo., 1. Juni 2026 um 16:44 Uhr schrieb Leonard Xu < > [email protected] > >> >: > >> > > >> >> Hi Fabian > >> >> > >> >> Thanks for driving this FLIP and your kind patience. The motivation > is > >> >> spot-on, and the LOAD→JOIN two-phase design is the right structural > >> fix for > >> >> the FLINK-19830 initialization problem. Overall direction +1 from my > >> side. > >> >> > >> >> Besides Jark’s idea about backlog and InputSelectable which may need > >> more > >> >> prerequisites, I’ve two concerns about current proposal: > >> >> > >> >> 1. LOAD phase backpressure. The FLIP assumes "seconds to a few > minutes" > >> >> for build-side init, but nothing enforces it. Large build-side tables > >> >> (e.g., 50M rows) + fast probe streams → unbuffered probe-side state > >> >> explosion. Should we add a config like max-buffer-size that applies > >> >> backpressure when exceeded or some metrics about buffer, rather than > >> >> silently piling up records? > >> >> > >> >> 2. Flip-point CPU spike. Joining all buffered probe records × > >> build-side > >> >> state in one shot differs fundamentally from event-time join's > >> incremental > >> >> watermark-batched emission. In the worst case this could cause a > >> >> TaskManager CPU spike and downstream shock. Worth considering > >> micro-batch > >> >> draining during flip transition? > >> >> > >> >> Looking forward to your thoughts. > >> >> > >> >> Best, > >> >> Leonard > >> >> > >> >>> 2026 6月 1 16:02，Fabian Hueske <[email protected]> 写道： > >> >>> > >> >>> Hi Leonard, > >> >>> > >> >>> Sorry, missed your email and already started the vote. > >> >>> Let me put it on hold for now and continue discussing the proposal. > >> >>> > >> >>> Looking forward to your comments, > >> >>> Fabian > >> >>> > >> >>> Am Mo., 1. Juni 2026 um 09:56 Uhr schrieb Leonard Xu < > >> [email protected] > >> >>> : > >> >>> > >> >>>> @Fabian Thanks for driving this FLIP, sorry for late reply due to > my > >> >>>> personal reason that I shouldn’t miss such an important FLIP. > >> >>>> > >> >>>> I’m reviewing the FLIP and will try to finish it today, could you > >> kindly > >> >>>> wait one minute to start the vote? > >> >>>> > >> >>>> And sorry for interrupt your plan again. > >> >>>> > >> >>>> Best, > >> >>>> Leonard > >> >>>> > >> >>>>> 2026 6月 1 15:51，Fabian Hueske <[email protected]> 写道： > >> >>>>> > >> >>>>> Thanks everyone for your comments on the FLIP. > >> >>>>> I will start the vote. > >> >>>>> > >> >>>>> Best, Fabian > >> >>>>> > >> >>>>> Am Do., 28. Mai 2026 um 20:13 Uhr schrieb David Anderson < > >> >>>>> [email protected]>: > >> >>>>> > >> >>>>>> Fabian, > >> >>>>>> > >> >>>>>>> So, I don't think that we should buffer unmatched probe-side > >> records > >> >>>>>> beyond > >> >>>>>> the flip point. > >> >>>>>> > >> >>>>>> Thanks for explaining your reasoning. Makes sense to me. > >> >>>>>> > >> >>>>>> David > >> >>>>>> > >> >>>>>> On Thu, May 28, 2026 at 6:55 PM Fabian Hueske <[email protected] > > > >> >>>> wrote: > >> >>>>>> > >> >>>>>>> Hi Xingcan, > >> >>>>>>> > >> >>>>>>> Thanks for your comments on the FLIP! > >> >>>>>>> > >> >>>>>>> The join's behavior when starting from a savepoint is indeed an > >> >>>> important > >> >>>>>>> aspect to consider and the problem of a rapidly advancing > >> dimension > >> >>>>>>> (build-side) table is of course real. > >> >>>>>>> > >> >>>>>>> I would argue that watermark alignment should significantly > reduce > >> >> the > >> >>>>>>> impact of this. > >> >>>>>>> If enabled, sources align their consumption based on their > current > >> >>>>>>> watermark such that the (presumably much smaller) build-side > >> source > >> >>>> would > >> >>>>>>> be slowed down to the event-time progress of the probe-side. > >> >>>>>>> While watermark alignment is not an "exact" mechanism, the > >> semantics > >> >> of > >> >>>>>> the > >> >>>>>>> new processing-time join also do not guarantee "exact" results. > >> >>>>>>> At the same time, alignment should ensure that build and > >> probe-side > >> >> are > >> >>>>>>> roughly aligned in event-time (without the strict guarantees > that > >> the > >> >>>>>>> event-time temporal table join provides). > >> >>>>>>> > >> >>>>>>> However, I really like your idea of starting in event-time mode > >> and > >> >>>>>>> flipping to processing-time after the initialization duration > >> passed. > >> >>>>>>> I'm not sure if it would fully address the problem you > described. > >> As > >> >>>> you > >> >>>>>>> said, users would need to be able to reconfigure the flip-point > >> and > >> >> I'm > >> >>>>>> not > >> >>>>>>> sure if there's a good mechanism for this yet. > >> >>>>>>> But it might have some other properties that would be > beneficial, > >> so > >> >>>> I'll > >> >>>>>>> think about that. > >> >>>>>>> > >> >>>>>>> Best, > >> >>>>>>> Fabian > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Am Do., 28. Mai 2026 um 18:21 Uhr schrieb Fabian Hueske < > >> >>>>>> [email protected] > >> >>>>>>>> : > >> >>>>>>> > >> >>>>>>>> Thanks for your feedback David! > >> >>>>>>>> > >> >>>>>>>>> One question: If I understand correctly, during the JOIN phase > >> of > >> >> an > >> >>>>>>>> INNER > >> >>>>>>>> join, if the desired build-side record is missing, nothing will > >> be > >> >>>>>>> emitted > >> >>>>>>>> for the unmatched probe-side record. For an INNER join, I can > >> >> imagine > >> >>>>>>>> wanting to buffer unmatched probe-side records, expecting the > >> build > >> >>>>>> side > >> >>>>>>>> will arrive soon. What's your thinking there? > >> >>>>>>>> > >> >>>>>>>> Your understanding is correct. If a probe-side record arrives > >> during > >> >>>>>> LOAD > >> >>>>>>>> phase but no matching build-side record is received, > >> >>>>>>>> the probe-side record would be discarded without being joined > >> during > >> >>>>>> the > >> >>>>>>>> transition from LOAD to JOIN. > >> >>>>>>>> > >> >>>>>>>> I would argue that users that want to prevent this, would need > to > >> >>>>>>>> configure a longer initialization time. > >> >>>>>>>> IMO, dropping unmatched probe records is not a "bad" property > of > >> >> INNER > >> >>>>>>>> joins but an essential part of their semantics. It might even > be > >> >>>>>> desired > >> >>>>>>> by > >> >>>>>>>> some users. > >> >>>>>>>> If we would buffer probe-side records for INNER joins beyond > the > >> >>>>>>>> transition point, we: > >> >>>>>>>> * would have different behaviors for INNER and LEFT joins > >> >>>>>>>> * could not start to emit probe-side watermarks as long as > there > >> are > >> >>>>>>> still > >> >>>>>>>> probe-side records buffered (or at least not advance past them > >> >> without > >> >>>>>>>> emitting late data at a later point of time) > >> >>>>>>>> * would either need another config knob to specify when to > >> "really" > >> >>>>>> clean > >> >>>>>>>> up the probe-side state or keep such unmatched records forever > in > >> >>>> state > >> >>>>>>> (we > >> >>>>>>>> could also use state TTL...) > >> >>>>>>>> > >> >>>>>>>> So, I don't think that we should buffer unmatched probe-side > >> records > >> >>>>>>>> beyond the flip point. > >> >>>>>>>> > >> >>>>>>>> Best, Fabian > >> >>>>>>>> > >> >>>>>>>> Am Do., 28. Mai 2026 um 17:05 Uhr schrieb Xingcan Cui < > >> >>>>>>> [email protected] > >> >>>>>>>>> : > >> >>>>>>>> > >> >>>>>>>>> Hi Fabian, > >> >>>>>>>>> > >> >>>>>>>>> Thanks for this FLIP! The two-phase design is excellent for > >> >> avoiding > >> >>>>>>>>> early-joining bugs while maintaining low-latency > processing-time > >> >>>>>>>>> semantics. > >> >>>>>>>>> > >> >>>>>>>>> After thinking more about the proposal, I'd like to point out > an > >> >> edge > >> >>>>>>> case > >> >>>>>>>>> related to the initialization phase or recovery after > prolonged > >> >>>>>> downtime > >> >>>>>>>>> (for example, when a job has been down for a day). While a > >> >>>>>>> processing-time > >> >>>>>>>>> join works well for live streaming, where results can > reasonably > >> >>>>>> depend > >> >>>>>>> on > >> >>>>>>>>> the immediate arrival order of live data, it does not work as > >> well > >> >>>> for > >> >>>>>>>>> catch-up scenarios. > >> >>>>>>>>> > >> >>>>>>>>> Currently, if a job initializes or restores from a checkpoint > >> >> after a > >> >>>>>>> long > >> >>>>>>>>> downtime, the operator resumes directly in the processing-time > >> join > >> >>>>>>> phase. > >> >>>>>>>>> During catch-up, however, the natural chronological arrival > >> order > >> >> of > >> >>>>>> the > >> >>>>>>>>> live data is completely lost. As a result, these replayed fact > >> >>>> records > >> >>>>>>> are > >> >>>>>>>>> evaluated against the current machine time and may blindly > join > >> >> with > >> >>>>>> the > >> >>>>>>>>> rapidly advancing "current" dimension snapshot, rather than > the > >> >>>>>>> historical > >> >>>>>>>>> versions they were originally supposed to match. > >> >>>>>>>>> > >> >>>>>>>>> To handle this edge case, could we consider: > >> >>>>>>>>> > >> >>>>>>>>> 1. changing the first phase into an event-time join phase, and > >> >>>>>>>>> > >> >>>>>>>>> 2. allowing the operator to switch back to the first phase > >> after a > >> >>>>>>>>> restart? > >> >>>>>>>>> > >> >>>>>>>>> For example, users could configure a timestamp threshold. > Before > >> >> the > >> >>>>>>>>> watermark reaches that point, the operator would run as an > >> >> event-time > >> >>>>>>>>> versioned join to safely process the catch-up phase through > >> >> watermark > >> >>>>>>>>> alignment. Once the watermark passes the threshold, the > operator > >> >>>> could > >> >>>>>>>>> purge the old multi-version state and seamlessly transition > >> back to > >> >>>>>> the > >> >>>>>>>>> pure processing-time join phase for live traffic. > >> >>>>>>>>> > >> >>>>>>>>> After a job restart, users could either update the target > >> timestamp > >> >>>> to > >> >>>>>>>>> reset the operator back into the event-time phase, or leave it > >> >>>>>> unchanged > >> >>>>>>>>> to > >> >>>>>>>>> continue operating in the processing-time phase. > >> >>>>>>>>> > >> >>>>>>>>> I completely understand that this would introduce significant > >> >>>>>> complexity > >> >>>>>>>>> to > >> >>>>>>>>> the operator's state management and lifecycle, so this is > only a > >> >>>>>>> tentative > >> >>>>>>>>> proposal to explore whether it might be worth considering for > >> the > >> >>>>>>>>> long-term > >> >>>>>>>>> robustness of the design. > >> >>>>>>>>> > >> >>>>>>>>> Best, > >> >>>>>>>>> > >> >>>>>>>>> Xingcan > >> >>>>>>>>> > >> >>>>>>>>> On Thu, May 28, 2026 at 8:17 AM David Anderson < > >> >> [email protected] > >> >>>>> > >> >>>>>>>>> wrote: > >> >>>>>>>>> > >> >>>>>>>>>> I'm quite enthusiastic about this. I want to thank Fabian for > >> >>>>>> putting > >> >>>>>>>>>> together such a well-crafted FLIP. And I look forward to > >> updating > >> >>>>>> the > >> >>>>>>>>>> awkward educational content this FLIP will make obsolete. > >> >>>>>>>>>> > >> >>>>>>>>>> To my mind, the syntax expresses the semantics of this join > >> rather > >> >>>>>>> well. > >> >>>>>>>>>> > >> >>>>>>>>>> Until now, developers using event-time temporal joins > sometimes > >> >>>>>>>>> resorted to > >> >>>>>>>>>> doing weird things with watermarks to handle a build side > >> that's > >> >>>>>>> mostly > >> >>>>>>>>>> idle; this lateral snapshot join is clearly better -- not to > >> >> mention > >> >>>>>>> the > >> >>>>>>>>>> added bonus of pre-loading the build table. > >> >>>>>>>>>> > >> >>>>>>>>>> One question: If I understand correctly, during the JOIN > phase > >> of > >> >> an > >> >>>>>>>>> INNER > >> >>>>>>>>>> join, if the desired build-side record is missing, nothing > >> will be > >> >>>>>>>>> emitted > >> >>>>>>>>>> for the unmatched probe-side record. For an INNER join, I can > >> >>>>>> imagine > >> >>>>>>>>>> wanting to buffer unmatched probe-side records, expecting the > >> >> build > >> >>>>>>> side > >> >>>>>>>>>> will arrive soon. What's your thinking there? > >> >>>>>>>>>> > >> >>>>>>>>>> David > >> >>>>>>>>>> > >> >>>>>>>>>> On Wed, May 27, 2026 at 12:44 PM Fabian Hueske < > >> >> [email protected]> > >> >>>>>>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>>> Thanks Gustavo and Timo for the positive feedback! > >> >>>>>>>>>>> > >> >>>>>>>>>>> I'd like to bump this thread up to collect more feedback. > >> >>>>>>>>>>> If there are no more responses, I will start a vote on this > >> FLIP > >> >>>>>>> next > >> >>>>>>>>>>> Monday, June 1st. > >> >>>>>>>>>>> > >> >>>>>>>>>>> Best, Fabian > >> >>>>>>>>>>> > >> >>>>>>>>>>> Am Do., 21. Mai 2026 um 12:15 Uhr schrieb Timo Walther < > >> >>>>>>>>>> [email protected] > >> >>>>>>>>>>>> : > >> >>>>>>>>>>> > >> >>>>>>>>>>>> Hi Fabian, > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> thanks for proposing this FLIP. I agree that this join is > >> super > >> >>>>>>>>> common, > >> >>>>>>>>>>>> after talking to many people at conferences, I could > imagine > >> it > >> >>>>>>>>> will be > >> >>>>>>>>>>>> one of the most used kinds of joins going forward. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Tightly coupling it with watermarks fits both from a > >> semantical > >> >>>>>>>>> point > >> >>>>>>>>>> of > >> >>>>>>>>>>>> view but also with other efforts such as FLIP-558 > >> (Improvements > >> >>>>>> to > >> >>>>>>>>>>>> SinkUpsertMaterializer and changelog disorder) [1]. In the > >> near > >> >>>>>>>>> future, > >> >>>>>>>>>>>> we should work on more automated watermarking to power > these > >> >>>>>>>>>>>> watermark-based operators, but this is an orthogonal > effort. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Overall I'm strongly +1 on this. Also +1 on the syntax > >> >>>>>>> improvements > >> >>>>>>>>> for > >> >>>>>>>>>>>> lateral table functions by dropping the TABLE() wrapper. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Cheers, > >> >>>>>>>>>>>> Timo > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> [1] > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>> > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-558%3A+Improvements+to+SinkUpsertMaterializer+and+changelog+disorder > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> On 18.05.26 11:47, Gustavo de Morais wrote: > >> >>>>>>>>>>>>> Hi Fabian, > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> In general a strong +1 for the feature, without getting > into > >> >>>>>> the > >> >>>>>>>>>>> details > >> >>>>>>>>>>>> of > >> >>>>>>>>>>>>> the FLIP yet. This is a missing feature for years and I'm > >> >>>>>> happy > >> >>>>>>>>> that > >> >>>>>>>>>>>> we're > >> >>>>>>>>>>>>> putting the time to address this - while also getting rid > of > >> >>>>>>> some > >> >>>>>>>>> of > >> >>>>>>>>>>> the > >> >>>>>>>>>>>>> hard restrictions we had. Thanks! > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Kind regards, > >> >>>>>>>>>>>>> Gustavo > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> On Fri, 15 May 2026 at 16:39, Fabian Hueske < > >> >>>>>> [email protected] > >> >>>>>>>> > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Hi everyone, > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> I'd like to start a discussion on FLIP-579: LATERAL > >> SNAPSHOT > >> >>>>>>> Join > >> >>>>>>>>>> [1]. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Enriching a stream with data from a (slowly changing) > >> dynamic > >> >>>>>>>>> table > >> >>>>>>>>>>> is a > >> >>>>>>>>>>>>>> super common use case. > >> >>>>>>>>>>>>>> Flink SQL features Temporal Joins [2] to address these > use > >> >>>>>>> cases. > >> >>>>>>>>>>>>>> However, SQL users can only use the event-time variant > >> which > >> >>>>>>> has > >> >>>>>>>>>> many > >> >>>>>>>>>>>>>> limitations (heavy dependency on frequent WM updates on > >> both > >> >>>>>>>>> inputs, > >> >>>>>>>>>>>>>> build-side table requires a PK, the join predicate must > >> >>>>>> include > >> >>>>>>>>> the > >> >>>>>>>>>>>>>> build-side PK, etc). > >> >>>>>>>>>>>>>> The processing-time temporal join is disabled (due to > >> >>>>>>> build-side > >> >>>>>>>>>>>>>> initialization issues [3]) and temporal table function > >> joins > >> >>>>>>> are > >> >>>>>>>>>>>>>> only available in Table API. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> FLIP-579 proposes a new temporal join operator that > >> operates > >> >>>>>> in > >> >>>>>>>>>>>>>> processing-time and addresses the limitations of the > >> existing > >> >>>>>>>>>>>>>> implementations: > >> >>>>>>>>>>>>>> * initialization of the build-side before joining > >> >>>>>>>>>>>>>> * no requirement of continuous, frequent build-side WMs > >> >>>>>> (after > >> >>>>>>>>> the > >> >>>>>>>>>>>>>> initialization completed) > >> >>>>>>>>>>>>>> * no requirement for a PK on the build-side > >> >>>>>>>>>>>>>> * table function-based syntax [4] via a built-in SNAPSHOT > >> >>>>>>>>> function > >> >>>>>>>>>>>>>> (proposed in FLIP-517 [4]) > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Looking forward to your feedback. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Best, > >> >>>>>>>>>>>>>> Fabian > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> [1] > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>> > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-579%3A+LATERAL+SNAPSHOT+Join > >> >>>>>>>>>>>>>> [2] > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>> > >> >> > >> > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/queries/joins/#temporal-joins > >> >>>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-19830 > >> >>>>>>>>>>>>>> [4] > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>> > >> >> > >> > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/sql/queries/joins/#temporal-table-function-join > >> >>>>>>>>>>>>>> [5] > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>> > >> >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-517%3A+Better+Handling+of+Dynamic+Table+Primitives+with+PTFs#FLIP517:BetterHandlingofDynamicTablePrimitiveswithPTFs-SNAPSHOTfortemporaljoins > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>> > >> >>>> > >> >> > >> >> > >> > >> >
