Thanks @Jiabao for kicking off this discussion. Could you add a section to explain the difference between proposed connector level config `scan.filter-push-down.enabled` and existing query level config `table.optimizer.source.predicate-pushdown-enabled` ?
Best, Leonard > 2023年10月24日 下午4:18，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> 写道： > > Hi Devs, > > I would like to start a discussion on FLIP-377: support configuration to > disable filter pushdown for Table/SQL Sources[1]. > > Currently, Flink Table/SQL does not expose fine-grained control for users to > enable or disable filter pushdown. > However, filter pushdown has some side effects, such as additional > computational pressure on external systems. > Moreover, Improper queries can lead to issues such as full table scans, which > in turn can impact the stability of external systems. > > Suppose we have an SQL query with two sources: Kafka and a database. > The database is sensitive to pressure, and we want to configure it to not > perform filter pushdown to the database source. > However, we still want to perform filter pushdown to the Kafka source to > decrease network IO. > > I propose to support configuration to disable filter push down for Table/SQL > sources to let user decide whether to perform filter pushdown. > > Looking forward to your feedback. > > [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 > > Best, > Jiabao
