Hi, Gustavo. Oh, I apologize for not noticing your reply before...
1. The quick answer is that as long as at least one streaming join is
transformed into a delta join, FORCE will not raise an error.
2. Related to FLIP-516, I am also gradually working on a POC for multi-way
delta join, but it ma
Hey Xuyang,
Happy to see the FLIP being approved. Could you reply to the questions
above if you have some time?
Thanks,
Gustavo
On Tue, 20 May 2025 at 08:56, Gustavo de Morais
wrote:
> Hey Xuyang,
>
> Thanks for proposing this and driving the discussion. In general, a very
> interesting idea.
Hey Xuyang,
Thanks for proposing this and driving the discussion. In general, a very
interesting idea.
- Can you go a bit in detail on how the optimizer will work in AUTO/FORCE
modes? When are we returning an error or falling back to the regular
streaming join operator? Do we support, for example
Hi, devs.
Due to the long duration of this discussion. ff there are no further
objections, I will start the voting thread tomorrow.
--
Best！
Xuyang
At 2025-05-19 10:05:46, "Xuyang" wrote:
>Hi, Xiangyu.
>
>You are correct that the lookup function on the lookup side should not u
Hi, Xiangyu.
You are correct that the lookup function on the lookup side should not use
caching. Therefore, we need to unwrap the actual lookup function from the
CachingLookupFunction(instead of updating it in place). We assume that the
cached function utilized by the lookup table is based on
Hi Xuyang,
Thx for driving this FLIP. I believe delta join is truly helpful for large
state scenarios. The overall design LGTM. Also, I like the idea of updating
the cache using the data from its own side.
I have one minor question: for storages like Paimon, it has its own cache
in its LookupFunc
Hi, Xuyang
Thanks for your reply, looks good to me.
Best,
Ron
Xuyang 于2025年5月13日周二 15:00写道：
> Hi, Feng. Let me address your questions:
>
> 1. As you mentioned, these filters should be recalculated. We can apply
> the filters after lookup the source,
>
> and then perform the join. Moreover, we
Hi, Feng. Let me address your questions:
1. As you mentioned, these filters should be recalculated. We can apply the
filters after lookup the source,
and then perform the join. Moreover, we can consider pushing the filters down
to the lookup source to
achieve a more efficient lookup.
2. You
Hi, xuyang
Thanks for proposing this FLIP — it’s very helpful for scenarios involving
large states.
I have a few questions regarding the node whitelist section:
1. If the TableSource implements FilterPushdownSpec, take LEFT JOIN as an
example — when the right side supports filter pushdown, the s
Hi, Ron. Thanks for your attention to this flip.
1. At first, inner/left/right/full join will be supported. I have updated the
flip about this part.
2. Are you referring to the situation where the downstream primary key (PK)
differs from the upstream join key? Once the sink requires the upstrea
Hi, Xuyang
Thanks for initiating this FLIP, which is of great value in solving the
Streaming Join that troubles many users. Big +1 for it.
For the overall design of FLIP, I have the following questions:
1. Can you explain the currently supported Join types in FLIP, such as
Inner, Left, and Right
Hi, Weijie.
Thanks for your review! Let me try to answer these questions.
1. The nodes between source and join must be included in the whitelist, such as
Calc,
DropUpdateBefore, WatermarkAssigner, Exchange and etc. You can see more in
chapter
"Limitations about Delta Join with Eventual Cons
Hi Xuyang,
Thanks for driving this! The state of two-stream join is a very
headache-inducing problem in stream computing system.
After reading this FLIP, I have two question:
1. Can the two sides of the join operator be any stream or must it be
immediately followed by the LookupSource? If it is
Hi, devs.
I'd like to start a discussion on FLIP-486: Introduce a new DeltaJoin[1].
In Flink streaming jobs, the large state of Join nodes has been a persistent
concern for users.
Since streaming jobs are long-running, the state of join generally increases in
size over time.
Although use
14 matches
Mail list logo
