Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-26 Thread godfrey he
Thanks for your confirmation. I will start the vote. Best, Godfrey Jing Zhang 于2022年7月26日周二 19:24写道: > > Hi, Godfrey > Thanks for updating the FLIP. > It looks good to me now. > > Best, > Jing Zhang > > godfrey he 于2022年7月26日周二 12:33写道: > > > Thanks for all the inputs, I have updated the docume

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-26 Thread Jing Zhang
Hi, Godfrey Thanks for updating the FLIP. It looks good to me now. Best, Jing Zhang godfrey he 于2022年7月26日周二 12:33写道: > Thanks for all the inputs, I have updated the document and POC code. > > > Best, > Godfrey > > Yun Gao 于2022年7月26日周二 11:11写道: > > > > Hi, > > > > Thanks all for all the valua

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-26 Thread Jingsong Li
Thanks for the update. Looks good to me. Best, Jingsong On Tue, Jul 26, 2022 at 3:21 PM Jark Wu wrote: > > Thank you Godfrey, the FLIP looks good to me. > > Best, > Jark > > On Tue, 26 Jul 2022 at 12:34, godfrey he wrote: > > > Thanks for all the inputs, I have updated the document and POC cod

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-26 Thread Jark Wu
Thank you Godfrey, the FLIP looks good to me. Best, Jark On Tue, 26 Jul 2022 at 12:34, godfrey he wrote: > Thanks for all the inputs, I have updated the document and POC code. > > > Best, > Godfrey > > Yun Gao 于2022年7月26日周二 11:11写道: > > > > Hi, > > > > Thanks all for all the valuable discussio

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-25 Thread godfrey he
Thanks for all the inputs, I have updated the document and POC code. Best, Godfrey Yun Gao 于2022年7月26日周二 11:11写道: > > Hi, > > Thanks all for all the valuable discussion on this FLIP, +1 for implementing > dynamic partition pruning / dynamic filtering pushdown since it is a key > optimization >

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-25 Thread Yun Gao
Hi, Thanks all for all the valuable discussion on this FLIP, +1 for implementing dynamic partition pruning / dynamic filtering pushdown since it is a key optimization to improve the performance on batch processing. Also due to introducing the speculative execution for the batch processing, we

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-21 Thread Jing Ge
Hi, Thanks for the informative discussion! Looking forward to using dynamic filtering provided by Flink. Best regards, Jing On Tue, Jul 19, 2022 at 3:22 AM godfrey he wrote: > Hi, Jingong, Jark, Jing, > > Thanks for for the important inputs. > Lake storage is a very important scenario, and con

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-18 Thread godfrey he
Hi, Jingong, Jark, Jing, Thanks for for the important inputs. Lake storage is a very important scenario, and consider more generic and extended case, I also would like to use "dynamic filtering" concept instead of "dynamic partition". >maybe the FLIP should also demonstrate the EXPLAIN result, wh

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-13 Thread Jing Zhang
Hi Godfrey, Thanks for driving this discussion. This is an important improvement for batch sql jobs. I agree with Jingsong to expand the capability to more than just partitions. Besides, I have two points: 1. Based on FLIP-248[1], > Dynamic partition pruning mechanism can improve performance by av

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-11 Thread Jark Wu
I agree with Jingsong. DPP is a particular case of Dynamic Filter Pushdown that the join key contains partition fields. Extending this FLIP to general filter pushdown can benefit more optimizations, and they can share the same interface. For example, Trino Hive Connector leverages dynamic filteri

Re: [DISCUSS] FLIP-248: Introduce dynamic partition pruning

2022-07-11 Thread Jingsong Li
Thanks Godfrey for driving. I like this FLIP. We can restrict this capability to more than just partitions. Here are some inputs from Lake Storage. The format of the splits generated by Lake Storage is roughly as follows: Split { Path filePath; Statistics[] fieldStats; } Stats contain the