Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-08 Thread Jing Ge
;> > > >> > for the implementation plan, mentioned in the FLIP that 1.18 > will > >> > > >> support Calc, HashJoin and HashAgg, then what will be the next > >> step? > >> > and > >> > > >> which operators do

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Aitozi
ding to their usage. Operators like Calc, > > > HashJoin, > > > > >> HashAgg, etc. are more commonly used, so we will support them > first. > > > > Later > > > > >> we support the rest of the operators step by step. Considering the &g

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread liu ron
in batch mode, but we >> > > >> prioritize them according to their usage. Operators like Calc, >> > HashJoin, >> > > >> HashAgg, etc. are more commonly used, so we will support them >> first. >> > > Later >> > > >> we support the rest

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread liu ron
t; >> Best, > > > >> Ron > > > >> > > > >> Jingsong Li 于2023年6月5日周一 14:15写道: > > > >> > > > >>> > For the state compatibility session, it seems that the checkpoint > > > >>> compatibility woul

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jing Ge
. Streaming should be > another > > >>> story? > > >>> > > >>> Best, > > >>> Jingsong > > >>> > > >>> On Mon, Jun 5, 2023 at 2:07 PM Yun Tang wrote: > > >>> > > > >>> > Hi R

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Aitozi
mprove the performance, looking > >>> forward to its completion in Flink! > >>> > > >>> > For the state compatibility session, it seems that the checkpoint > >>> compatibility would be broken just like [1] did. Could FLIP-190 [2] > still >

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-06 Thread liu ron
For the state compatibility session, it seems that the checkpoint >>> compatibility would be broken just like [1] did. Could FLIP-190 [2] still >>> be helpful in this case for SQL version upgrades? >>> > >>> > >>> > [1] >>> https://docs.google.com/document/d/1qKV

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-05 Thread liu ron
UOqVHsI/edit#heading=h.fri5rtpte0si >> > [2] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 >> > >> > Best >> > Yun Tang >> > >> > >> > From: Lincoln Lee >> > Se

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-05 Thread liu ron
t; > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 > > > > Best > > Yun Tang > > > > ________________ > > From: Lincoln Lee > > Sent: Monday, June 5, 2023 10:56 > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS]

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-04 Thread Benchao Li
pte0si > > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 > > > > Best > > Yun Tang > > > > ____________ > > From: Lincoln Lee > > Sent: Monday, June 5, 2023 10:56 > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS] FLIP

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-04 Thread Jingsong Li
iewpage.action?pageId=191336489 > > Best > Yun Tang > > > From: Lincoln Lee > Sent: Monday, June 5, 2023 10:56 > To: dev@flink.apache.org > Subject: Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL > > Hi Ron > > OFGC lo

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-04 Thread Yun Tang
To: dev@flink.apache.org Subject: Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL Hi Ron OFGC looks like an exciting optimization, looking forward to its completion in Flink! A small question, do we consider adding a benchmark for the operators to intuitively understand the

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-04 Thread Lincoln Lee
Hi Ron OFGC looks like an exciting optimization, looking forward to its completion in Flink! A small question, do we consider adding a benchmark for the operators to intuitively understand the improvement brought by each improvement? In addition, for the implementation plan, mentioned in the FLIP

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-04 Thread liu ron
Hi, Jark Thanks for your feedback, according to my initial assessment, the work effort is relatively large. Moreover, I will add a test result of all queries to the FLIP. Best, Ron Jark Wu 于2023年6月1日周四 20:45写道: > Hi Ron, > > Thanks a lot for the great proposal. The FLIP looks good to me in ge

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-01 Thread Jark Wu
Hi Ron, Thanks a lot for the great proposal. The FLIP looks good to me in general. It looks like not an easy work but the performance sounds promising. So I think it's worth doing. Besides, if there is a complete test graph with all TPC-DS queries, the effect of this FLIP will be more intuitive.

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-05-30 Thread liu ron
Hi, Jinsong Thanks for your valuable suggestions. Best, Ron Jingsong Li 于2023年5月30日周二 13:22写道: > Thanks Ron for your information. > > I suggest that it can be written in the Motivation of FLIP. > > Best, > Jingsong > > On Tue, May 30, 2023 at 9:57 AM liu ron wrote: > > > > Hi, Jingsong > > >

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-05-29 Thread Jingsong Li
Thanks Ron for your information. I suggest that it can be written in the Motivation of FLIP. Best, Jingsong On Tue, May 30, 2023 at 9:57 AM liu ron wrote: > > Hi, Jingsong > > Thanks for your review. We have tested it in TPC-DS case, and got a 12% > gain overall when only supporting only Calc&H

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-05-29 Thread liu ron
Hi, Jingsong Thanks for your review. We have tested it in TPC-DS case, and got a 12% gain overall when only supporting only Calc&HashJoin&HashAgg operator. In some queries, we even get more than 30% gain, it looks like an effective way. Best, Ron Jingsong Li 于2023年5月29日周一 14:33写道: > Thanks Ro

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-05-28 Thread Jingsong Li
Thanks Ron for the proposal. Do you have some benchmark results for the performance improvement? I am more concerned about the improvement on Flink than the data in other papers. Best, Jingsong On Mon, May 29, 2023 at 2:16 PM liu ron wrote: > > Hi, dev > > I'd like to start a discussion about F