Re: [DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-03-10 Thread Jeyhun Karimov
Hi Martijn, Thanks for your comment. I created a quick prototype to make sure that the proposed hypothesis works. So, it still is a quick implementation. On my local setup all tests pass, but I might need to spend some time (once VOTE thread passes probably) to clean it and submit a PR. Each test

Re: [DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-03-08 Thread Martijn Visser
Hi Jeyhun Karimov, I see that you've already opened up a VOTE thread, but since you're talking about having a prototype already and results, I wondered if you could include the POC and how you've tested these results in the FLIP? Best regards, Martijn On Tue, Jan 30, 2024 at 4:47 AM Jeyhun Kari

Re: [DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-01-30 Thread Jeyhun Karimov
Hi devs, I just wanted to give an update on this FLIP. I updated the doc based on the comments from Jim. Also, I developed a prototype and did some testing. I in my small prototype I ran the following tests: - org.apache.flink.table.planner.plan.stream.sql.DagOptimizationTest#testMultiSin

Re: [DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-01-17 Thread Jeyhun Karimov
Hi Jim, Thanks for your comments. Please find my answers below: 1. StreamOptimizeContext may still be needed to pass the fact that we >are optimizing a streaming query. I don't think this class will go away >completely. (I agree it may become more simple if the kind or >mini-batc

Re: [DISCUSS] FLIP-419: Optimize multi-sink query plan generation

2024-01-16 Thread Jim Hughes
Hi Jeyhun, Generally, I like the idea of speeding up the optimizer in the case of multiple queries! I am new to the optimizer, but I have a few comments / questions. 1. StreamOptimizeContext may still be needed to pass the fact that we are optimizing a streaming query. I don't think t