Thanks Xuyang, If we only add configuration without adding the enableFilterPushDown method in the SupportsFilterPushDown interface, each connector would have to handle the same logic in the applyFilters method to determine whether filter pushdown is needed. This would increase complexity and violate the original behavior of the applyFilters method.
On the contrary, we only need to pass the configuration parameter in the newly added enableFilterPushDown method to decide whether to perform predicate pushdown. I think this approach would be clearer and simpler. Best, Jiabao > 2023年10月24日 14:34，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> 写道： > > Thanks Xuyang, > > The table.optimizer.source.predicate-pushdown-enabled options do not provide > fine-grained configuration for each source. > > Suppose we have an SQL query with two sources: Kafka and a database (CDC). > The database is sensitive to pressure, and we want to configure it to not > perform filter pushdown to the database source. > However, we still want to perform filter pushdown to the Kafka source to > decrease network IO. > > > Best, > Jiabao > <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/table/config/#table-optimizer-source-predicate-pushdown-enabled> > >> 2023年10月24日 14:24，Xuyang <xyzhong...@163.com> 写道： >> >> Hi, the existant configuration >> 'table.optimizer.source.predicate-pushdown-enabled' seems to do what you >> want. >> Can you describe more clearly the difference between what you want and this >> configuration ? >> >> >> >> >> >> >> >> -- >> >> Best！ >> Xuyang >> >> >> >> >> >> At 2023-10-24 14:12:14, "Jiabao Sun" <jiabao....@xtransfer.cn.INVALID> wrote: >>> Hi Devs, >>> >>> I would like to start a discussion on support configuration to disable >>> filter pushdown for Table/SQL Sources[1]. >>> >>> Currently, Flink SQL does not support the ability for users to enable or >>> disable filter pushdown. >>> However, filter pushdown has some side effects, such as additional >>> computational pressure on external systems. >>> Moreover, Improper queries can lead to issues such as full table scans, >>> which in turn can impact the stability of external systems. >>> >>> I propose to support configuration to disable filter push down for >>> Table/SQL sources to let user decide whether to perform filter pushdown. >>> >>> Looking forward to your feedback. >>> >>> [1] >>> https://docs.google.com/document/d/1QsbOi9InvmfwFr8YbrnnXOKLPnb8JnqhXIMbGd68SFU/edit?usp=sharing >>> >>> Best, >>> Jiabao >
