Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-24 Thread Jiabao Sun
Thanks Martijn and Xuyang, I opened a new discussion thread about FLIP-377. https://lists.apache.org/thread/nvxx8sp9jm009yywm075hoffr632tm7j Best, Jiabao > 2023年10月24日 15:39,Martijn Visser 写道: > > Hi, > > Please convert the Google Doc into a FLIP and start a FLIP discussion. > > Best regard

Re: Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-24 Thread Martijn Visser
Hi, Please convert the Google Doc into a FLIP and start a FLIP discussion. Best regards, Martijn On Tue, Oct 24, 2023 at 9:20 AM Xuyang wrote: > > +1. How to set the configuration value so that the specific source can be > perceived needs to be considered. > > > > > -- > > Best! > Xuy

Re:Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-24 Thread Xuyang
+1. How to set the configuration value so that the specific source can be perceived needs to be considered. -- Best! Xuyang At 2023-10-24 15:05:03, "Jiabao Sun" wrote: >Thanks Xuyang, > >If we only add configuration without adding the enableFilterPushDown method in >the Support

Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-24 Thread Jiabao Sun
Thanks Xuyang, If we only add configuration without adding the enableFilterPushDown method in the SupportsFilterPushDown interface, each connector would have to handle the same logic in the applyFilters method to determine whether filter pushdown is needed. This would increase complexity and v

Re:Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-23 Thread Xuyang
Thanks for you explain. It seems that what you actually need is a general connector parameter in 'WITH' block like 'filter-pushdown-enabled', and the specific connector rejects all filters that could otherwise be pushed down in the connector. Please forgive me if I'm wrong. -- Best!

Re: [DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-23 Thread Jiabao Sun
Thanks Xuyang, The table.optimizer.source.predicate-pushdown-enabled options do not provide fine-grained configuration for each source. Suppose we have an SQL query with two sources: Kafka and a database (CDC). The database is sensitive to pressure, and we want to configure it to not perform f

Re:[DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-23 Thread Xuyang
Hi, the existant configuration 'table.optimizer.source.predicate-pushdown-enabled' seems to do what you want. Can you describe more clearly the difference between what you want and this configuration ? -- Best! Xuyang At 2023-10-24 14:12:14, "Jiabao Sun" wrote: >Hi Devs, >

[DISCUSS] Support configuration to disable filter pushdown for Table/SQL Sources

2023-10-23 Thread Jiabao Sun
Hi Devs, I would like to start a discussion on support configuration to disable filter pushdown for Table/SQL Sources[1]. Currently, Flink SQL does not support the ability for users to enable or disable filter pushdown. However, filter pushdown has some side effects, such as additional comput