Hi, Thank you to everyone for the discussion on this FLIP, especially Becket for providing guidance that made it more reasonable.
The FLIP document[1] has been updated with the recent discussed content. Please take a look to double-check it when you have time. If we can reach a consensus on this, I will open the voting thread in recent days. Best, Jiabao [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 > 2023年12月20日 11:38，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> 写道： > > Thanks Becket, > > The behavior description has been added to the Public Interfaces section. > > Best, > Jiabao > > >> 2023年12月20日 08:17，Becket Qin <becket....@gmail.com> 写道： >> >> Hi Jiabao, >> >> Thanks for updating the FLIP. >> Can you add the behavior of the policies that are only applicable to some >> but not all of the databases? This is a part of the intended behavior of >> the proposed configuration. So, we should include that in the FLIP. >> Otherwise, the FLIP looks good to me. >> >> Cheers, >> >> Jiangjie (Becket) Qin >> >> On Tue, Dec 19, 2023 at 11:00 PM Jiabao Sun <jiabao....@xtransfer.cn.invalid> >> wrote: >> >>> Hi Becket, >>> >>> I share the same view as you regarding the prefix for this configuration >>> option. >>> >>> For the JDBC connector, I prefer setting 'filter.handling.policy' = 'FOO' >>> and throwing an exception when the database do not support that specific >>> policy. >>> >>> Not using a prefix can reduce the learning curve for users and avoid >>> introducing a new set of configuration options for every supported JDBC >>> database. >>> I think the policies we provide can be compatible with most databases that >>> follow the JDBC protocol. >>> However, there may be cases where certain databases cannot support some >>> policies. >>> Nevertheless, we can ensure fast failure and allow users to choose a >>> suitable policy in such situations. >>> >>> I have removed the contents about the configuration prefix. >>> Please help review it again. >>> >>> Thanks, >>> Jiabao >>> >>> >>>> 2023年12月19日 19:46，Becket Qin <becket....@gmail.com> 写道： >>>> >>>> Hi Jiabao, >>>> >>>> Thanks for updating the FLIP. >>>> >>>> One more question regarding the JDBC connector, since it is a connector >>>> shared by multiple databases, what if there is a filter handling policy >>>> that is only applicable to one of the databases, but not the others? In >>>> that case, how would the users specify that policy? >>>> Unlike the example of orc format with 2nd+ level config, JDBC connector >>>> only looks at the URL to decide which driver to use. >>>> >>>> For example, MySql supports policy FOO while other databases do not. If >>>> users want to use FOO for MySql, what should they do? Will they set >>>> '*mysql.filter.hanlding.policy' >>>> = 'FOO', *which will only be picked up when the MySql driver is used? >>>> Or they should just set* 'filter.handling.policy' = 'FOO', *and throw >>>> exceptions when other JDBC drivers are used? Personally, I prefer the >>>> latter. If we pick that, do we still need to mention the following? >>>> >>>> *The prefix is needed when the option is for a 2nd+ level. * >>>>> 'connector' = 'filesystem', >>>>> 'format' = 'orc', >>>>> 'orc.filter.handling.policy' = 'NUBERIC_ONY' >>>>> >>>>> *In this case, the values of this configuration may be different >>> depending >>>>> on the format option. For example, orc format may have INDEXED_ONLY >>> while >>>>> parquet format may have something else. * >>>>> >>>> >>>> I found this is somewhat misleading, because the example here is not a >>> part >>>> of the proposal of this FLIP. It is just an example explaining when a >>>> prefix is needed, which seems orthogonal to the proposal in this FLIP. >>>> >>>> Thanks, >>>> >>>> Jiangjie (Becket) Qin >>>> >>>> >>>> On Tue, Dec 19, 2023 at 10:09 AM Jiabao Sun <jiabao....@xtransfer.cn >>> .invalid> >>>> wrote: >>>> >>>>> Thanks Becket for the suggestions, >>>>> >>>>> Updated. >>>>> Please help review it again when you have time. >>>>> >>>>> Best, >>>>> Jiabao >>>>> >>>>> >>>>>> 2023年12月19日 09:06，Becket Qin <becket....@gmail.com> 写道： >>>>>> >>>>>> Hi JIabao, >>>>>> >>>>>> Thanks for updating the FLIP. It looks better. Some suggestions / >>>>> questions: >>>>>> >>>>>> 1. In the motivation section: >>>>>> >>>>>>> *Currently, Flink Table/SQL does not expose fine-grained control for >>>>> users >>>>>>> to control filter pushdown. **However, filter pushdown has some side >>>>>>> effects, such as additional computational pressure on external >>>>>>> systems. Moreover, Improper queries can lead to issues such as full >>>>> table >>>>>>> scans, which in turn can impact the stability of external systems.* >>>>>> >>>>>> This statement sounds like the side effects are there for all the >>>>> systems, >>>>>> which is inaccurate. Maybe we can say: >>>>>> *Currently, Flink Table/SQL does not expose fine-grained control for >>>>> users >>>>>> to control filter pushdown. **However, filter pushdown may have side >>>>>> effects in some cases, **such as additional computational pressure on >>>>>> external systems. The JDBC source is a typical example of that. If a >>>>> filter >>>>>> is pushed down to the database, an expensive full table scan may happen >>>>> if >>>>>> the filter involves unindexed columns.* >>>>>> >>>>>> 2. Regarding the prefix, usually a prefix is not required for the top >>>>> level >>>>>> connector options. This is because the *connector* option is already >>>>> there. >>>>>> So >>>>>> 'connector' = 'jdbc', >>>>>> 'filter.handling.policy' = 'ALWAYS' >>>>>> is sufficient. >>>>>> >>>>>> The prefix is needed when the option is for a 2nd+ level. For example, >>>>>> 'connector' = 'jdbc', >>>>>> 'format' = 'orc', >>>>>> 'orc.some.option' = 'some_value' >>>>>> In this case, the prefix of "orc" is needed to make it clear this >>> option >>>>> is >>>>>> for the format. >>>>>> >>>>>> I am guessing that the reason that currently the connector prefix is >>>>> there >>>>>> is because the values of this configuration may be different depending >>> on >>>>>> the connectors. For example, jdbc may have INDEXED_ONLY while MongoDB >>> may >>>>>> have something else. Personally speaking, I am fine if we do not have a >>>>>> prefix in this case because users have already specified the connector >>>>> type >>>>>> and it is intuitive enough that the option value is for that connector, >>>>> not >>>>>> others. >>>>>> >>>>>> 3. can we clarify on the following statement: >>>>>> >>>>>>> *Introduce the native configuration [prefix].filter.handling.policy in >>>>> the >>>>>>> connector.* >>>>>> >>>>>> What do you mean by "native configuration"? From what I understand, the >>>>>> FLIP does the following: >>>>>> - introduces a new configuration to the JDBC and MongoDB connector. >>>>>> - Suggests a convention option name if other connectors are going to >>> add >>>>> an >>>>>> option for the same purpose. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jiangjie (Becket) Qin >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Dec 18, 2023 at 5:45 PM Jiabao Sun <jiabao....@xtransfer.cn >>>>> .invalid> >>>>>> wrote: >>>>>> >>>>>>> Hi Becket, >>>>>>> >>>>>>> The FLIP document[1] has been updated. >>>>>>> Could you help take a look again? >>>>>>> >>>>>>> Thanks, >>>>>>> Jiabao >>>>>>> >>>>>>> [1] >>>>>>> >>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 >>>>>>> >>>>>>> >>>>>>>> 2023年12月18日 16:53，Becket Qin <becket....@gmail.com> 写道： >>>>>>>> >>>>>>>> Yes, that sounds reasonable to me. We can start with ALWAYS and >>> NEVER, >>>>>>> and >>>>>>>> add more policies as needed. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Jiangjie (Becket) Qin >>>>>>>> >>>>>>>> On Mon, Dec 18, 2023 at 4:48 PM Jiabao Sun <jiabao....@xtransfer.cn >>>>>>> .invalid> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks Bucket, >>>>>>>>> >>>>>>>>> The jdbc.filter.handling.policy is good to me as it provides >>>>> sufficient >>>>>>>>> extensibility for future filter pushdown optimizations. >>>>>>>>> However, currently, we don't have an implementation for the AUTO >>> mode, >>>>>>> and >>>>>>>>> it seems that the AUTO mode can easily be confused with the ALWAYS >>>>> mode >>>>>>>>> because users don't have the opportunity to MANUALLY decide which >>>>>>> filters >>>>>>>>> to push down. >>>>>>>>> >>>>>>>>> I suggest that we only introduce the ALWAYS and NEVER modes for now, >>>>> and >>>>>>>>> we can consider introducing more flexible policies in the future, >>>>>>>>> such as INDEX_ONLY, NUMBERIC_ONLY and so on. >>>>>>>>> >>>>>>>>> WDYT? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Jiabao >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> 2023年12月18日 16:27，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>> >>>>>>>>>> Hi Jiabao, >>>>>>>>>> >>>>>>>>>> Please see the reply inline. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> The MySQL connector is currently in the flink-connector-jdbc >>>>>>> repository >>>>>>>>>>> and is not a standalone connector. >>>>>>>>>>> Is it too unique to use "mysql" as the configuration option >>> prefix? >>>>>>>>>> >>>>>>>>>> If the intended behavior makes sense to all the supported JDBC >>>>> drivers, >>>>>>>>> we >>>>>>>>>> can make this a JDBC connector configuration. >>>>>>>>>> >>>>>>>>>> Also, I would like to ask about the difference in behavior between >>>>> AUTO >>>>>>>>> and >>>>>>>>>>> ALWAYS. >>>>>>>>>>> It seems that we cannot guarantee the pushing down of all filters >>> to >>>>>>> the >>>>>>>>>>> external system under the ALWAYS >>>>>>>>>>> mode because not all filters in Flink SQL are supported by the >>>>>>> external >>>>>>>>>>> system. >>>>>>>>>>> Should we throw an error when encountering a filter that cannot be >>>>>>>>> pushed >>>>>>>>>>> down in the ALWAYS mode? >>>>>>>>>> >>>>>>>>>> The idea of AUTO is to do efficiency-aware pushdowns. The source >>> will >>>>>>>>> query >>>>>>>>>> the external system (MySQL, Oracle, SQL Server, etc) first to >>>>> retrieve >>>>>>>>> the >>>>>>>>>> information of the table. With that information, the source will >>>>> decide >>>>>>>>>> whether to further push a filter to the external system based on >>> the >>>>>>>>>> efficiency. E.g. only push the indexed fields. In contrast, ALWAYS >>>>> will >>>>>>>>>> just always push the supported filters to the external system, >>>>>>> regardless >>>>>>>>>> of the efficiency. In case there are filters that are not >>> supported, >>>>>>>>>> according to the current contract of SupportsFilterPushdown, these >>>>>>>>>> unsupported filters should just be returned by the >>>>>>>>>> *SupportsFilterPushdown.applyFilters()* method as remaining >>> filters. >>>>>>>>>> Therefore, there is no need to throw exceptions here. This is >>> likely >>>>>>> the >>>>>>>>>> desired behavior for most users, IMO. If there are cases that users >>>>>>>>> really >>>>>>>>>> want to get alerted when a filter cannot be pushed to the external >>>>>>>>> system, >>>>>>>>>> we can add another value like "ENFORCED_ALWAYS", which behaves like >>>>>>>>> ALWAYS, >>>>>>>>>> but throws exceptions when a filter cannot be applied to the >>> external >>>>>>>>>> system. But personally I don't see much value in doing this. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Dec 18, 2023 at 3:54 PM Jiabao Sun < >>> jiabao....@xtransfer.cn >>>>>>>>> .invalid> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Becket, >>>>>>>>>>> >>>>>>>>>>> The MySQL connector is currently in the flink-connector-jdbc >>>>>>> repository >>>>>>>>>>> and is not a standalone connector. >>>>>>>>>>> Is it too unique to use "mysql" as the configuration option >>> prefix? >>>>>>>>>>> >>>>>>>>>>> Also, I would like to ask about the difference in behavior between >>>>>>> AUTO >>>>>>>>>>> and ALWAYS. >>>>>>>>>>> It seems that we cannot guarantee the pushing down of all filters >>> to >>>>>>> the >>>>>>>>>>> external system under the ALWAYS >>>>>>>>>>> mode because not all filters in Flink SQL are supported by the >>>>>>> external >>>>>>>>>>> system. >>>>>>>>>>> Should we throw an error when encountering a filter that cannot be >>>>>>>>> pushed >>>>>>>>>>> down in the ALWAYS mode? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Jiabao >>>>>>>>>>> >>>>>>>>>>>> 2023年12月18日 15:34，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>> >>>>>>>>>>>> Hi JIabao, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for updating the FLIP. Maybe I did not explain it clearly >>>>>>>>> enough. >>>>>>>>>>> My >>>>>>>>>>>> point is that given there are various good flavors of behaviors >>>>>>>>> handling >>>>>>>>>>>> filters pushed down, we should not have a common config of >>>>>>>>>>>> "ignore.filter.pushdown", because the behavior is not *common*. >>>>>>>>>>>> >>>>>>>>>>>> It looks like the original motivation of this FLIP is just for >>>>> MySql. >>>>>>>>>>> Let's >>>>>>>>>>>> focus on what is the best solution for MySql connector here >>> first. >>>>>>>>> After >>>>>>>>>>>> that, if people think the best behavior for MySql happens to be a >>>>>>>>> common >>>>>>>>>>>> one, we can then discuss whether that is worth being added to the >>>>>>> base >>>>>>>>>>>> implementation of source. For MySQL, if we are going to >>> introduce a >>>>>>>>>>> config >>>>>>>>>>>> to MySql, why not have something like >>>>> "mysql.filter.handling.policy" >>>>>>>>> with >>>>>>>>>>>> value of AUTO / NEVER / ALWAYS? Isn't that better than >>>>>>>>>>>> "ignore.filter.pushdown"? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Dec 17, 2023 at 11:30 PM Jiabao Sun < >>>>> jiabao....@xtransfer.cn >>>>>>>>>>> .invalid> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>> >>>>>>>>>>>>> The FLIP document has been updated as well. >>>>>>>>>>>>> Please take a look when you have time. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Jiabao >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> 2023年12月17日 22:54，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> >>>>> 写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Becket, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I apologize for not being able to continue with this proposal >>> due >>>>>>> to >>>>>>>>>>>>> being too busy during this period. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The viewpoints you shared about the design of Flink Source make >>>>>>> sense >>>>>>>>>>> to >>>>>>>>>>>>> me >>>>>>>>>>>>>> The native configuration ‘ignore.filter.pushdown’ is good to >>> me. >>>>>>>>>>>>>> Having a unified name or naming style can indeed prevent >>>>> confusion >>>>>>>>> for >>>>>>>>>>>>> users regarding >>>>>>>>>>>>>> the inconsistent naming of this configuration across different >>>>>>>>>>>>> connectors. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Currently, there are not many external connectors that support >>>>>>> filter >>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>> I propose that we first introduce it in flink-connector-jdbc >>> and >>>>>>>>>>>>> flink-connector-mongodb. >>>>>>>>>>>>>> Do you think this is feasible? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2023年11月16日 17:45，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Arguments like "because Spark has it so Flink should also have >>>>> it" >>>>>>>>>>> does >>>>>>>>>>>>> not >>>>>>>>>>>>>>> make sense. Different projects have different API flavors and >>>>>>>>> styles. >>>>>>>>>>>>> What >>>>>>>>>>>>>>> is really important is the rationale and the design principle >>>>>>> behind >>>>>>>>>>> the >>>>>>>>>>>>>>> API. They should conform to the convention of the project. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> First of all, Spark Source API itself has a few issues and >>> they >>>>>>>>> ended >>>>>>>>>>> up >>>>>>>>>>>>>>> introduce DataSource V2 in Spark 3.0, which added the >>> decorative >>>>>>>>>>>>> interfaces >>>>>>>>>>>>>>> like SupportsPushdownXXX. Some of the configurations predating >>>>>>>>>>>>> DataSource >>>>>>>>>>>>>>> V2 may still be there. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For the Spark configurations you mentioned, they are all the >>>>>>>>>>>>> configurations >>>>>>>>>>>>>>> for FileScanBuilder, which is equivalent to FileSource in >>> Flink. >>>>>>>>>>>>> Currently, >>>>>>>>>>>>>>> regardless of the format (ORC, Parquet, Avro, etc), the >>>>> FileSource >>>>>>>>>>>>> pushes >>>>>>>>>>>>>>> back all the filters to ensure correctness. The actual filters >>>>>>> that >>>>>>>>>>> got >>>>>>>>>>>>>>> applied to the specific format might still be different. This >>>>>>>>>>>>>>> implementation is the same in FileScanBuilder.pushFilters() >>> for >>>>>>>>>>> Spark. I >>>>>>>>>>>>>>> don't know why Spark got separate configurations for each >>>>> format. >>>>>>>>>>> Maybe >>>>>>>>>>>>> it >>>>>>>>>>>>>>> is because the filters are actually implemented differently >>> for >>>>>>>>>>>>> different >>>>>>>>>>>>>>> format. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> At least for the current implementation in FileScanBuilder, >>>>> these >>>>>>>>>>>>>>> configurations can be merged to one configuration like >>>>>>>>>>>>>>> `apply.filters.to.format.enabled`. Note that this config, as >>>>> well >>>>>>> as >>>>>>>>>>> the >>>>>>>>>>>>>>> separate configs you mentioned, are just visible and used by >>> the >>>>>>>>>>>>>>> FileScanBuilder. It determines whether the filters should be >>>>>>> passed >>>>>>>>>>>>> down to >>>>>>>>>>>>>>> the format of the FileScanBuilder instance. Regardless of the >>>>>>> value >>>>>>>>> of >>>>>>>>>>>>>>> these configs, FileScanBuilder.pushFilters() will always be >>>>>>> called, >>>>>>>>>>> and >>>>>>>>>>>>>>> FileScanBuilder (as well as FileSource in Flink) will always >>>>> push >>>>>>>>> back >>>>>>>>>>>>> all >>>>>>>>>>>>>>> the filters to the framework. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> A MySql source can have a very different way to handle this. >>> For >>>>>>>>>>>>> example, A >>>>>>>>>>>>>>> MySql source A config in this case might be >>> "my.apply.filters" >>>>>>> with >>>>>>>>>>>>> three >>>>>>>>>>>>>>> different values: >>>>>>>>>>>>>>> - AUTO: The Source will issue a DESC Table request to >>> understand >>>>>>>>>>>>> whether a >>>>>>>>>>>>>>> filter can be applied efficiently. And decide which filters >>> can >>>>> be >>>>>>>>>>>>> applied >>>>>>>>>>>>>>> and which cannot based on that. >>>>>>>>>>>>>>> - NEVER: Never apply filtering. It will always do a full table >>>>>>> read >>>>>>>>>>> and >>>>>>>>>>>>>>> let Flink do the filtering. >>>>>>>>>>>>>>> - ALWAYS: Always apply the filtering to the MySql server. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In the above examples of FileSource and MySql Source, I don't >>>>>>> think >>>>>>>>> it >>>>>>>>>>>>> is a >>>>>>>>>>>>>>> good idea to shoehorn the behaviors into a naive config of >>>>>>>>>>>>>>> `ignore.filter.pushdown`. That is why I don't think this is a >>>>>>> common >>>>>>>>>>>>> config. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To recap, like I said, I do agree that in some cases, we may >>>>> want >>>>>>> to >>>>>>>>>>>>> behave >>>>>>>>>>>>>>> differently when filters are pushed down to the sources, even >>>>> if a >>>>>>>>>>>>> source >>>>>>>>>>>>>>> implements SupportsFilterPushDown, but I don't think there is >>> a >>>>>>>>>>> suitable >>>>>>>>>>>>>>> common config for this. The behavior is very likely source >>>>>>> specific. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Nov 16, 2023 at 3:41 PM Jiabao Sun < >>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Becket, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I still believe that adding a configuration at the source >>> level >>>>>>> to >>>>>>>>>>>>> disable >>>>>>>>>>>>>>>> filter pushdown is needed. This demand exists in spark as >>>>>>> well[1]. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In Spark, most sources that support filter pushdown provide >>>>> their >>>>>>>>> own >>>>>>>>>>>>>>>> corresponding configuration options to enable or disable >>> filter >>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>> For PRs[2-4] that support filter pushdown capability, they >>> also >>>>>>>>>>> provide >>>>>>>>>>>>>>>> configuration options to disable this capability. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I believe this configuration is applicable to most scenarios, >>>>> and >>>>>>>>>>>>> there is >>>>>>>>>>>>>>>> no need to dwell on why this configuration option was not >>>>>>>>> introduced >>>>>>>>>>>>>>>> earlier than the SupportsFilterPushDown interface. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> spark.sql.parquet.filterPushdown >>>>>>>>>>>>>>>> spark.sql.orc.filterPushdown >>>>>>>>>>>>>>>> spark.sql.csv.filterPushdown.enabled >>>>>>>>>>>>>>>> spark.sql.json.filterPushdown.enabled >>>>>>>>>>>>>>>> spark.sql.avro.filterPushdown.enabled >>>>>>>>>>>>>>>> JDBC Option: pushDownPredicate >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We can see that the lack of consistency is caused by each >>>>>>> connector >>>>>>>>>>>>>>>> introducing different configuration options for the same >>>>>>> behavior. >>>>>>>>>>>>>>>> This is one of the motivations for advocating the >>> introduction >>>>>>> of a >>>>>>>>>>>>>>>> unified configuration name. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/SPARK-24288 >>>>>>>>>>>>>>>> [2] https://github.com/apache/spark/pull/27366 >>>>>>>>>>>>>>>> [3]https://github.com/apache/spark/pull/26973 >>>>>>>>>>>>>>>> [4] https://github.com/apache/spark/pull/29145 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2023年11月16日 08:10，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> While we can always fix the formality of the config, a more >>>>>>>>>>>>> fundamental >>>>>>>>>>>>>>>>> issue here is whether this configuration is common enough. >>>>>>>>>>> Personally >>>>>>>>>>>>> I >>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>> still not convinced it is. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Remember we don't have a common implementation for >>>>>>>>>>>>> SupportsFilterPushdown >>>>>>>>>>>>>>>>> itself. Why does a potential behavior of the >>>>>>>>>>>>>>>>> SupportsFilterPushdown.applyFilters() method deserve a >>> common >>>>>>>>>>>>>>>>> configuration? A common implementation should always come >>>>> first, >>>>>>>>>>> then >>>>>>>>>>>>> its >>>>>>>>>>>>>>>>> configuration becomes a common configuration as a natural >>>>>>> result. >>>>>>>>>>> But >>>>>>>>>>>>>>>> here >>>>>>>>>>>>>>>>> we are trying to add an impl to a configuration just to fix >>>>> its >>>>>>>>>>>>>>>> formality. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I agree that there might be a few Source implementations >>> that >>>>>>> may >>>>>>>>>>>>> want to >>>>>>>>>>>>>>>>> avoid additional burdens on the remote system in some >>>>>>>>> circumstances. >>>>>>>>>>>>> And >>>>>>>>>>>>>>>>> these circumstances are very specific: >>>>>>>>>>>>>>>>> 1. The source talks to a remote service that can help >>> perform >>>>>>> the >>>>>>>>>>>>> actual >>>>>>>>>>>>>>>>> filtering. >>>>>>>>>>>>>>>>> 2. The filtering done by the remote service is inefficient >>> for >>>>>>>>> some >>>>>>>>>>>>>>>> reason >>>>>>>>>>>>>>>>> (e.g. missing index) >>>>>>>>>>>>>>>>> 3. The external service does not want to perform the >>>>> inefficient >>>>>>>>>>>>>>>> filtering >>>>>>>>>>>>>>>>> for some reason (e.g. it is a shared service with others) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> There are multiple approaches to address the issue. Pushing >>>>> back >>>>>>>>> the >>>>>>>>>>>>>>>>> filters is just one way of achieving this. So here we are >>>>>>> talking >>>>>>>>>>>>> about a >>>>>>>>>>>>>>>>> config for one of the possible solutions to a scenario with >>>>> all >>>>>>>>> the >>>>>>>>>>>>> above >>>>>>>>>>>>>>>>> situations. I don't think there is enough justification for >>>>> the >>>>>>>>>>>>> config to >>>>>>>>>>>>>>>>> be common. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> There is always this trade-off between the proliferation of >>>>>>> public >>>>>>>>>>>>>>>>> interfaces and the API standardization. As an extreme >>> example, >>>>>>> we >>>>>>>>>>> can >>>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>>> our public API a union of all the configs potentially used >>> in >>>>>>> all >>>>>>>>>>> the >>>>>>>>>>>>>>>> cases >>>>>>>>>>>>>>>>> in the name of standardization. Apparently this won't work. >>> So >>>>>>>>> there >>>>>>>>>>>>> must >>>>>>>>>>>>>>>>> be a bar here and this bar might be somewhat subjective. For >>>>>>> this >>>>>>>>>>>>> FLIP, >>>>>>>>>>>>>>>>> personally I don't think the config meets my bar for the >>>>> reason >>>>>>>>>>> stated >>>>>>>>>>>>>>>>> above. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Therefore, my suggestion remains the same. Keep the config >>> as >>>>> a >>>>>>>>>>> Source >>>>>>>>>>>>>>>>> implementation specific configuration. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Nov 16, 2023 at 12:36 AM Jiabao Sun < >>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks Becket for the feedback, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regarding concerns about common configurations, I think we >>>>> can >>>>>>>>>>>>> introduce >>>>>>>>>>>>>>>>>> FiltersApplier to unify the behavior of various connectors. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> public static class FiltersApplier { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> private final ReadableConfig config; >>>>>>>>>>>>>>>>>> private final Function<List<ResolvedExpression>, Result> >>>>>>> action; >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> private FiltersApplier( >>>>>>>>>>>>>>>>>> ReadableConfig config, >>>>>>>>>>>>>>>>>> Function<List<ResolvedExpression>, Result> action) { >>>>>>>>>>>>>>>>>> this.config = config; >>>>>>>>>>>>>>>>>> this.action = action; >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> public Result applyFilters(List<ResolvedExpression> >>> filters) >>>>> { >>>>>>>>>>>>>>>>>> if (config.get(ENABLE_FILTER_PUSH_DOWN)) { >>>>>>>>>>>>>>>>>> return action.apply(filters); >>>>>>>>>>>>>>>>>> } else { >>>>>>>>>>>>>>>>>> return Result.of(Collections.emptyList(), filters); >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> public static FiltersApplier of( >>>>>>>>>>>>>>>>>> ReadableConfig config, >>>>>>>>>>>>>>>>>> Function<List<ResolvedExpression>, Result> action) { >>>>>>>>>>>>>>>>>> return new FiltersApplier(config, action); >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For connectors implementation: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> @Override >>>>>>>>>>>>>>>>>> public Result applyFilters(List<ResolvedExpression> >>> filters) >>>>> { >>>>>>>>>>>>>>>>>> return FiltersApplier.of(config, >>>>>>>>>>>>>>>>>> f -> Result.of(new ArrayList<>(filters), >>>>>>>>>>>>>>>>>> Collections.emptyList())); >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> As for the name, whether it is >>>>>>> "source.filter-push-down.enabled" >>>>>>>>> or >>>>>>>>>>>>>>>>>> "source.ignore-pushed-down-filters.enabled", I think both >>> are >>>>>>>>> okay. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Do you think this change is feasible? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2023年11月15日 23:44，Becket Qin <becket....@gmail.com> 写道： >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Yes, I still have concerns. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The FLIP violates the following two principles regarding >>>>>>>>>>>>> configuration: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1.* A config of a class should never negate the semantic >>> of >>>>> a >>>>>>>>>>>>>>>> decorative >>>>>>>>>>>>>>>>>>> interface implemented by that class. * >>>>>>>>>>>>>>>>>>> A decorative interface is a public contract with other >>>>>>>>> components, >>>>>>>>>>>>>>>> while >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> config is only internal to the class itself. The >>>>>>> configurations >>>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> Sources are not (and should never be) visible or >>> understood >>>>> to >>>>>>>>>>>>>>>>>>> other components (e.g. optimizer). A configuration of a >>>>> Source >>>>>>>>>>> only >>>>>>>>>>>>>>>>>>> controls the behavior of that Source, provided it is not >>>>>>>>> violating >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> API >>>>>>>>>>>>>>>>>>> contract / semantic defined by the decorative interface. >>> So >>>>>>>>> when a >>>>>>>>>>>>>>>> Source >>>>>>>>>>>>>>>>>>> implementation implements SupportsFilterPushdown, this is >>> a >>>>>>>>> clear >>>>>>>>>>>>>>>> public >>>>>>>>>>>>>>>>>>> contract with Flink that filters should be pushed down to >>>>> that >>>>>>>>>>>>> Source. >>>>>>>>>>>>>>>>>>> Therefore, for the same source, there should not be a >>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>> "source.filter-push-down.enabled" which stops the filters >>>>> from >>>>>>>>>>> being >>>>>>>>>>>>>>>>>> pushed >>>>>>>>>>>>>>>>>>> down to that Source. However, that specific source >>>>>>>>> implementation >>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>> its own config to control its internal behavior, e.g. >>>>>>>>>>>>>>>>>>> "ignore-pushed-down-filters.enabled" which may push back >>> all >>>>>>> the >>>>>>>>>>>>> pushed >>>>>>>>>>>>>>>>>>> down filters back to the Flink optimizer. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2. When we are talking about "common configs", in fact we >>>>> are >>>>>>>>>>>>> talking >>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>> "configs for common (abstract) implementation classes". >>> With >>>>>>>>> that >>>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>> context, *a common config should always be backed by a >>>>> common >>>>>>>>>>>>>>>>>>> implementation class, so that consistent behavior can be >>>>>>>>>>>>> guaranteed. * >>>>>>>>>>>>>>>>>>> The LookupOptions you mentioned are configurations defined >>>>> for >>>>>>>>>>>>> classes >>>>>>>>>>>>>>>>>>> DefaultLookupCache / PeriodicCacheReloadTrigger / >>>>>>>>>>>>>>>>>> TimedCacheReloadTrigger. >>>>>>>>>>>>>>>>>>> These configs are considered as "common" only because the >>>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>>>>> classes using them are common building blocks for lookup >>>>> table >>>>>>>>>>>>>>>>>>> implementations. It would not make sense to have a >>> dangling >>>>>>>>> config >>>>>>>>>>>>> in >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> LookupOptions without the underlying common implementation >>>>>>>>> class, >>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>> only >>>>>>>>>>>>>>>>>>> relies on a specific source to implement the stated >>>>> behavior. >>>>>>>>>>>>>>>>>>> As a bad example, there is this outlier config >>> "max-retries" >>>>>>> in >>>>>>>>>>>>>>>>>>> LookupOptions, which I don't think should be here. This is >>>>>>>>> because >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> retry behavior can be very implementation specific. For >>>>>>> example, >>>>>>>>>>>>> there >>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> be many different flavors of retry related configurations, >>>>>>>>>>>>>>>> retry-backoff, >>>>>>>>>>>>>>>>>>> retry-timeout, retry-async, etc. Why only max-retry is put >>>>>>> here? >>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>> of them be put here? If we put all such kinds of configs >>> in >>>>>>> the >>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>> configs for "standardization and unification", the number >>> of >>>>>>>>>>> "common >>>>>>>>>>>>>>>>>>> configs" can easily go crazy. And I don't see material >>>>>>> benefits >>>>>>>>> of >>>>>>>>>>>>>>>> doing >>>>>>>>>>>>>>>>>>> that. So here I don't think the configuration "max-retry" >>>>>>> should >>>>>>>>>>> be >>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> LookupOptions, because it is not backed by any common >>>>>>>>>>> implementation >>>>>>>>>>>>>>>>>>> classes. If max-retry is implemented in the HBase source, >>> it >>>>>>>>>>> should >>>>>>>>>>>>>>>> stay >>>>>>>>>>>>>>>>>>> there. For the same reason, the config proposed in this >>> FLIP >>>>>>>>>>>>> (probably >>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> a name less confusing for the first reason mentioned >>> above) >>>>>>>>>>> should >>>>>>>>>>>>>>>> stay >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> the specific Source implementation. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> For the two reasons above, I am -1 to what the FLIP >>>>> currently >>>>>>>>>>>>> proposes. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I think the right way to address the motivation here is >>>>> still >>>>>>> to >>>>>>>>>>>>> have a >>>>>>>>>>>>>>>>>>> config like "ignore-pushed-down-filters.enabled" for the >>>>>>>>> specific >>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>> implementation. Please let me know if this solves the >>>>> problem >>>>>>>>> you >>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>> facing. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Wed, Nov 15, 2023 at 11:52 AM Jiabao Sun < >>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The purpose of introducing this configuration is that not >>>>> all >>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>> pushdowns can improve overall performance. >>>>>>>>>>>>>>>>>>>> If the filter can hit the external index, then pushdown >>> is >>>>>>>>>>>>> definitely >>>>>>>>>>>>>>>>>>>> worth it, as it can not only improve query time but also >>>>>>>>> decrease >>>>>>>>>>>>>>>>>> network >>>>>>>>>>>>>>>>>>>> overhead. >>>>>>>>>>>>>>>>>>>> However, for filters that do not hit the external index, >>> it >>>>>>> may >>>>>>>>>>>>>>>>>> increase a >>>>>>>>>>>>>>>>>>>> lot of performance overhead on the external system. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Undeniably, if the connector can make accurate decisions >>>>> for >>>>>>>>> good >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> bad >>>>>>>>>>>>>>>>>>>> filters, we may not need to introduce this configuration >>>>>>> option >>>>>>>>>>> to >>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>> pushing down filters to the external system. >>>>>>>>>>>>>>>>>>>> However, it is currently not easy to achieve. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> IMO, supporting filter pushdown does not mean that always >>>>>>>>> filter >>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>> is better. >>>>>>>>>>>>>>>>>>>> In the absence of automatic decision-making, I think we >>>>>>> should >>>>>>>>>>>>> leave >>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> decision to users. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The newly introduced configuration option is similar to >>>>>>>>>>>>> LookupOptions, >>>>>>>>>>>>>>>>>>>> providing unified naming and default values to avoid >>>>>>> confusion >>>>>>>>>>>>> caused >>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>> inconsistent naming in different connectors for users. >>>>>>>>>>>>>>>>>>>> Setting the default value to true allows it to maintain >>>>>>>>>>>>> compatibility >>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>> the default behavior of "always pushdown". >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Do you have any other concerns about this proposal? >>> Please >>>>>>> let >>>>>>>>> me >>>>>>>>>>>>>>>> know. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> 2023年10月31日 17:29，Jiabao Sun <jiabao....@xtransfer.cn >>>>>>>>> .INVALID> >>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Actually, for FileSystemSource, it is not always >>> desired, >>>>>>> only >>>>>>>>>>> OCR >>>>>>>>>>>>>>>> file >>>>>>>>>>>>>>>>>>>> formats support filter pushdown. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> We can disable predicate pushdown for FileSystemSource >>> by >>>>>>>>>>> setting >>>>>>>>>>>>>>>>>>>> 'table.optimizer.source.predicate-pushdown-enabled' to >>>>> false. >>>>>>>>>>>>>>>>>>>>> I think we can also disable filter pushdown at a more >>>>>>> granular >>>>>>>>>>>>> level >>>>>>>>>>>>>>>>>>>> through fine-grained configuration. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 16:50，Becket Qin <becket....@gmail.com> >>> 写道： >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the explanation. Maybe it's easier to >>> explain >>>>>>> with >>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>> example. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Let's take FileSystemTableSource as an example. >>> Currently >>>>>>> it >>>>>>>>>>>>>>>>>> implements >>>>>>>>>>>>>>>>>>>>>> SupportsFilterPushDown interface. With your proposal, >>>>> does >>>>>>> it >>>>>>>>>>>>> have >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> support `source.filter-push-down.enabled` as well? But >>>>> this >>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>> does not quite make sense for the FileSystemTableSource >>>>>>>>> because >>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>> pushdown is always desired. However, because this >>>>>>>>> configuration >>>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>>> part >>>>>>>>>>>>>>>>>>>>>> of the SupportsFilterPushDown interface (which sounds >>>>>>>>> confusing >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> begin >>>>>>>>>>>>>>>>>>>>>> with), the FileSystemTableSource can only do one of the >>>>>>>>>>>>> following: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. Ignore the user configuration to always apply the >>>>> pushed >>>>>>>>>>> down >>>>>>>>>>>>>>>>>>>> filters - >>>>>>>>>>>>>>>>>>>>>> this is an apparent anti-pattern because a >>> configuration >>>>>>>>> should >>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>>> what it says. >>>>>>>>>>>>>>>>>>>>>> 2. Throw an exception telling users that this >>>>> configuration >>>>>>>>> is >>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>> applicable to the FileSystemTableSource. >>>>>>>>>>>>>>>>>>>>>> 3. Implement this configuration to push back the pushed >>>>>>> down >>>>>>>>>>>>>>>> filters, >>>>>>>>>>>>>>>>>>>> even >>>>>>>>>>>>>>>>>>>>>> though this is never desired. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> None of the above options looks awkward. I am curious >>>>> what >>>>>>>>> your >>>>>>>>>>>>>>>>>>>> solution is >>>>>>>>>>>>>>>>>>>>>> here? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 31, 2023 at 3:11 PM Jiabao Sun < >>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks Becket for the further explanation. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Perhaps I didn't explain it clearly. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 1. If a source does not implement the >>>>>>> SupportsFilterPushDown >>>>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>> the newly added configurations do not need to be added >>>>> to >>>>>>>>>>> either >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> requiredOptions or optionalOptions. >>>>>>>>>>>>>>>>>>>>>>> Similar to LookupOptions, if a source does not >>> implement >>>>>>>>>>>>>>>>>>>>>>> LookupTableSource, there is no need to add >>> LookupOptions >>>>>>> to >>>>>>>>>>>>> either >>>>>>>>>>>>>>>>>>>>>>> requiredOptions or optionalOptions. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 2. "And these configs are specific to those sources, >>>>>>> instead >>>>>>>>>>> of >>>>>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>>> configs." >>>>>>>>>>>>>>>>>>>>>>> The newly introduced configurations define >>> standardized >>>>>>>>> names >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>> values. >>>>>>>>>>>>>>>>>>>>>>> They still belong to the configuration at the >>> individual >>>>>>>>>>> source >>>>>>>>>>>>>>>>>> level. >>>>>>>>>>>>>>>>>>>>>>> The purpose is to avoid scattered configuration items >>>>> when >>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>> sources implement the same logic. >>>>>>>>>>>>>>>>>>>>>>> Whether a source should accept these configurations is >>>>>>>>>>>>> determined >>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> source's Factory. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 13:47，Becket Qin <becket....@gmail.com> >>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Please see the replies inline. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Introducing common configurations does not mean that >>>>> all >>>>>>>>>>>>> sources >>>>>>>>>>>>>>>>>> must >>>>>>>>>>>>>>>>>>>>>>>>> accept these configuration options. >>>>>>>>>>>>>>>>>>>>>>>>> The configuration options supported by a source are >>>>>>>>>>>>> determined by >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> requiredOptions and optionalOptions in the Factory >>>>>>>>>>> interface. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> This is not true. Both required and optional options >>>>> are >>>>>>>>>>>>>>>> SUPPORTED. >>>>>>>>>>>>>>>>>>>> That >>>>>>>>>>>>>>>>>>>>>>>> means they are implemented and if one specifies an >>>>>>> optional >>>>>>>>>>>>> config >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>> still take effect. An OptionalConfig is "Optional" >>>>>>> because >>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>> configuration has a default value. Hence, it is OK >>> that >>>>>>>>> users >>>>>>>>>>>>> do >>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>> specify their own value. In another word, it is >>>>>>> "optional" >>>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> end >>>>>>>>>>>>>>>>>>>>>>>> users to set the config, but the implementation and >>>>>>> support >>>>>>>>>>> for >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>> config >>>>>>>>>>>>>>>>>>>>>>>> is NOT optional. In case a source does not support a >>>>>>> common >>>>>>>>>>>>>>>> config, >>>>>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>> exception must be thrown when the config is provided >>> by >>>>>>> the >>>>>>>>>>> end >>>>>>>>>>>>>>>>>> users. >>>>>>>>>>>>>>>>>>>>>>>> However, the config we are talking about in this FLIP >>>>> is >>>>>>> a >>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>> config >>>>>>>>>>>>>>>>>>>>>>>> optional to implement, meaning that sometimes the >>>>> claimed >>>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>> won't >>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>> there even if users specify that config. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Similar to sources that do not implement the >>>>>>>>>>> LookupTableSource >>>>>>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>>>> sources that do not implement the >>>>> SupportsFilterPushDown >>>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>>>>>> not need to accept newly introduced options. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> First of all, filter pushdown is a behavior of the >>>>> query >>>>>>>>>>>>>>>> optimizer, >>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> behavior of Sources. The Sources tells the optimizer >>>>> that >>>>>>>>> it >>>>>>>>>>>>> has >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> ability to accept pushed down filters by implementing >>>>> the >>>>>>>>>>>>>>>>>>>>>>>> SupportsFilterPushDown interface. And this is the >>> only >>>>>>>>>>> contract >>>>>>>>>>>>>>>>>>>> between >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>> Source and Optimizer regarding whether filters should >>>>> be >>>>>>>>>>> pushed >>>>>>>>>>>>>>>>>> down. >>>>>>>>>>>>>>>>>>>> As >>>>>>>>>>>>>>>>>>>>>>>> long as a specific source implements this decorative >>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>> pushdown should always take place, i.e. >>>>>>>>>>>>>>>>>>>>>>>> *SupportsFilterPushDown.applyFilters()* will be >>> called. >>>>>>>>> There >>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>> other config to disable that call. However, Sources >>> can >>>>>>>>>>> decide >>>>>>>>>>>>> how >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>> behave based on their own configurations after >>>>>>>>>>>>> *applyFilters()* is >>>>>>>>>>>>>>>>>>>>>>> called. >>>>>>>>>>>>>>>>>>>>>>>> And these configs are specific to those sources, >>>>> instead >>>>>>> of >>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>>> configs. >>>>>>>>>>>>>>>>>>>>>>>> Please see the examples I mentioned in the previous >>>>>>> email. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 31, 2023 at 10:27 AM Jiabao Sun < >>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Sorry, there was a typo in the second point. Let me >>>>>>>>> correct >>>>>>>>>>>>> it: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Introducing common configurations does not mean that >>>>> all >>>>>>>>>>>>> sources >>>>>>>>>>>>>>>>>> must >>>>>>>>>>>>>>>>>>>>>>>>> accept these configuration options. >>>>>>>>>>>>>>>>>>>>>>>>> The configuration options supported by a source are >>>>>>>>>>>>> determined by >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> requiredOptions and optionalOptions in the Factory >>>>>>>>>>> interface. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Similar to sources that do not implement the >>>>>>>>>>> LookupTableSource >>>>>>>>>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>>>> sources that do not implement the >>>>> SupportsFilterPushDown >>>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>> do >>>>>>>>>>>>>>>>>>>>>>>>> not need to accept newly introduced options. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 10:13，Jiabao Sun < >>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>> .INVALID> >>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Becket for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Currently, the >>> SupportsFilterPushDown#applyFilters >>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>> returns a >>>>>>>>>>>>>>>>>>>>>>>>> result that includes acceptedFilters and >>>>>>> remainingFilters. >>>>>>>>>>> The >>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>> decide to push down some filters or not accept any >>> of >>>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introducing common configuration options does >>> not >>>>>>> mean >>>>>>>>>>>>> that a >>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>> that supports the SupportsFilterPushDown capability >>>>> must >>>>>>>>>>>>> accept >>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>> configuration. Similar to LookupOptions, only >>> sources >>>>>>> that >>>>>>>>>>>>>>>>>> implement >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> LookupTableSource interface are necessary to accept >>>>>>> these >>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>> options. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月31日 07:49，Becket Qin < >>> becket....@gmail.com> >>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao and Ruanhang, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Adding a configuration of >>>>>>>>> source.filter-push-down.enabled >>>>>>>>>>>>> as a >>>>>>>>>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>>>>>>> source configuration seems problematic. >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The config name is misleading. filter pushdown >>>>>>> should >>>>>>>>>>>>> only >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>> determined >>>>>>>>>>>>>>>>>>>>>>>>>>> by whether the SupportsFilterPushdown interface is >>>>>>>>>>>>> implemented >>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>> not. >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. The behavior of this configuration is only >>>>>>> applicable >>>>>>>>>>> to >>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>> implementations. Why is it a common configuration? >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Here's my suggestion for design principles: >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Only add source impl specific configuration to >>>>>>>>>>>>> corresponding >>>>>>>>>>>>>>>>>>>>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. The configuration name should not overrule >>>>> existing >>>>>>>>>>>>> common >>>>>>>>>>>>>>>>>>>>>>> contracts. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> For example, in the case of MySql source. There >>> are >>>>>>>>>>> several >>>>>>>>>>>>>>>>>>>> options: >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Have a configuration of >>>>>>>>>>>>>>>>>> `*mysql.avoid.remote.full.table.scan`*. >>>>>>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration is set, and a filter pushdown does >>> not >>>>>>> hit >>>>>>>>>>> an >>>>>>>>>>>>>>>>>> index, >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> MySql source impl would not further pushdown the >>>>>>> filter >>>>>>>>> to >>>>>>>>>>>>>>>> MySql >>>>>>>>>>>>>>>>>>>>>>>>> servers. >>>>>>>>>>>>>>>>>>>>>>>>>>> Note that this assumes the MySql source can >>> retrieve >>>>>>> the >>>>>>>>>>>>> index >>>>>>>>>>>>>>>>>>>>>>>>> information >>>>>>>>>>>>>>>>>>>>>>>>>>> from the MySql servers. >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. If the MySql index information is not available >>>>> to >>>>>>>>> the >>>>>>>>>>>>> MySql >>>>>>>>>>>>>>>>>>>>>>> source, >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration could be something like >>>>>>>>>>>>>>>>>>>>>>>>> *`mysql.pushback.pushed.down.filters`*. >>>>>>>>>>>>>>>>>>>>>>>>>>> Once set to true, MySql source would just add all >>>>> the >>>>>>>>>>>>> filters >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> RemainingFilters in the Result returned by >>>>>>>>>>>>>>>>>>>>>>>>>>> *SupportsFilterPushdown.applyFilters().* >>>>>>>>>>>>>>>>>>>>>>>>>>> 3. An alternative to option 2 is to have a ` >>>>>>>>>>>>>>>>>>>>>>>>>>> *mysql.apply.predicates.after.scan*`. When it is >>> set >>>>>>> to >>>>>>>>>>>>> true, >>>>>>>>>>>>>>>>>> MySql >>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>> will not push the filter down to the MySql >>> servers, >>>>>>> but >>>>>>>>>>>>> apply >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> filters >>>>>>>>>>>>>>>>>>>>>>>>>>> inside the MySql source itself. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> As you may see, the above configurations do not >>>>>>> disable >>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>>>>>> itself. They just allow various implementations of >>>>>>>>> filter >>>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>> And >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration name does not give any illusion that >>>>>>>>> filter >>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>> disabled. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Oct 30, 2023 at 11:58 PM Jiabao Sun < >>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Hang for the suggestion. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think the configuration of TableSource is not >>>>>>> closely >>>>>>>>>>>>>>>> related >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>> SourceReader, >>>>>>>>>>>>>>>>>>>>>>>>>>>> so I prefer to introduce a independent >>>>> configuration >>>>>>>>>>> class >>>>>>>>>>>>>>>>>>>>>>>>>>>> TableSourceOptions in the flink-table-common >>>>> module, >>>>>>>>>>>>> similar >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>> LookupOptions. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> For the second point, I suggest adding Java doc >>> to >>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> SupportsXXXPushDown >>>>>>>>>>>>>>>>>>>>>>>>>>>> interfaces, providing detailed information on >>> these >>>>>>>>>>> options >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>> needs >>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>> be supported. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I have made updates in the FLIP document. >>>>>>>>>>>>>>>>>>>>>>>>>>>> Please help check it again. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月30日 17:23，Hang Ruan < >>>>> ruanhang1...@gmail.com >>>>>>>> >>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the improvements, Jiabao. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> There are some details that I am not sure about. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The new option >>>>> `source.filter-push-down.enabled` >>>>>>>>> will >>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>> added to >>>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>>> class? I think it should be >>> `SourceReaderOptions`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. How are the connector developers able to know >>>>> and >>>>>>>>>>>>> follow >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> FLIP? >>>>>>>>>>>>>>>>>>>>>>>>> Do >>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>> need an abstract base class or provide a default >>>>>>>>> method? >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hang >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao Sun <jiabao....@xtransfer.cn.invalid> >>>>>>>>>>>>> 于2023年10月30日周一 >>>>>>>>>>>>>>>>>>>>>>> 14:45写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, all, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the lively discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Based on the discussion, I have made some >>>>>>> adjustments >>>>>>>>>>> to >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> FLIP >>>>>>>>>>>>>>>>>>>>>>>>>>>> document: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The name of the newly added option has been >>>>>>>>> changed >>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "source.filter-push-down.enabled". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Considering compatibility with older >>> versions, >>>>>>> the >>>>>>>>>>>>> newly >>>>>>>>>>>>>>>>>>>> added >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "source.filter-push-down.enabled" option needs >>> to >>>>>>>>>>> respect >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> optimizer's >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>> option. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But there is a consideration to remove the old >>>>>>> option >>>>>>>>>>> in >>>>>>>>>>>>>>>> Flink >>>>>>>>>>>>>>>>>>>> 2.0. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. We can provide more options to disable other >>>>>>>>> source >>>>>>>>>>>>>>>>>> abilities >>>>>>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>>>>>> side >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effects, such as “source.aggregate.enabled” and >>>>>>>>>>>>>>>>>>>>>>>>>>>> “source.projection.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is not urgent and can be continuously >>>>>>>>> introduced. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to your feedback again. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月29日 08:45，Becket Qin < >>>>>>> becket....@gmail.com> >>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for digging into the git history, >>> Jark. I >>>>>>>>> agree >>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> makes >>>>>>>>>>>>>>>>>>>>>>>>> sense >>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate this API in 2.0. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Oct 27, 2023 at 5:47 PM Jark Wu < >>>>>>>>>>>>> imj...@gmail.com> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Becket, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I checked the history of " >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> *table.optimizer.source.predicate-pushdown-enabled*", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it seems it was introduced since the legacy >>>>>>>>>>>>>>>>>>>> FilterableTableSource >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> interface >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which might be an experiential feature at >>> that >>>>>>>>> time. >>>>>>>>>>> I >>>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessity >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of this option at the moment. Maybe we can >>>>>>>>> deprecate >>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> option >>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> drop >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Flink 2.0[1] if it is not necessary >>> anymore. >>>>>>>>> This >>>>>>>>>>>>> may >>>>>>>>>>>>>>>>>> help >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> simplify this discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jark >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1]: >>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-32383 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, 26 Oct 2023 at 10:14, Becket Qin < >>>>>>>>>>>>>>>>>>>> becket....@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the proposal, Jiabao. My two >>> cents >>>>>>>>> below: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. If I understand correctly, the motivation >>>>> of >>>>>>>>> the >>>>>>>>>>>>> FLIP >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> mainly to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make predicate pushdown optional on SOME of >>>>> the >>>>>>>>>>>>> Sources. >>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>> so, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intuitively >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the configuration should be Source specific >>>>>>>>> instead >>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>> general. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we will end up with general configurations >>>>> that >>>>>>>>> may >>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>>>>>>>>> effect >>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> some of the Source implementations. This >>>>>>> violates >>>>>>>>>>> the >>>>>>>>>>>>>>>> basic >>>>>>>>>>>>>>>>>>>> rule >>>>>>>>>>>>>>>>>>>>>>>>> of a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration - it does what it says, >>>>> regardless >>>>>>>>> of >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> While configuration standardization is >>>>> usually a >>>>>>>>>>> good >>>>>>>>>>>>>>>>>> thing, >>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> break the basic rules. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we really want to have this general >>>>>>>>>>> configuration, >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> sources >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this configuration does not apply, they >>> should >>>>>>>>> throw >>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>> exception >>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it clear that this configuration is not >>>>>>> supported. >>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>> seems >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ugly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I think the actual motivation of this >>> FLIP >>>>> is >>>>>>>>>>> about >>>>>>>>>>>>>>>>>> "how a >>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should implement predicate pushdown >>>>>>> efficiently", >>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>> "whether >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown should be applied to the source." >>> For >>>>>>>>>>>>> example, >>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wants >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to avoid additional computing load in the >>>>>>> external >>>>>>>>>>>>>>>> system, >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> read the entire record and apply the >>>>> predicates >>>>>>> by >>>>>>>>>>>>>>>> itself. >>>>>>>>>>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Flink perspective, the predicate >>> pushdown >>>>> is >>>>>>>>>>>>> applied, >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implemented differently by the source. So >>> the >>>>>>>>> design >>>>>>>>>>>>>>>>>>>> principle >>>>>>>>>>>>>>>>>>>>>>>>> here >>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Flink only cares about whether a source >>>>> supports >>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it does not care about the implementation >>>>>>>>>>> efficiency / >>>>>>>>>>>>>>>> side >>>>>>>>>>>>>>>>>>>>>>>>> effect of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicates pushdown. It is the Source >>>>>>>>>>> implementation's >>>>>>>>>>>>>>>>>>>>>>>>> responsibility >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ensure the predicates pushdown is >>> implemented >>>>>>>>>>>>> efficiently >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>> does >>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> impose excessive pressure on the external >>>>>>> system. >>>>>>>>>>> And >>>>>>>>>>>>> it >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>> OK >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> additional configurations to achieve this >>>>> goal. >>>>>>>>>>>>>>>> Obviously, >>>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations will be source specific in >>> this >>>>>>>>> case. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Regarding the existing configurations of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> *table.optimizer.source.predicate-pushdown-enabled. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *I am not sure why we need it. Supposedly, >>> if >>>>> a >>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>> implements >>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SupportsXXXPushDown interface, the optimizer >>>>>>>>> should >>>>>>>>>>>>> push >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> corresponding >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicates to the Source. I am not sure in >>>>> which >>>>>>>>>>> case >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would be used. Any ideas @Jark Wu < >>>>>>>>> imj...@gmail.com >>>>>>>>>>>> ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 25, 2023 at 11:55 PM Jiabao Sun >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <jiabao....@xtransfer.cn.invalid> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jane for the detailed explanation. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think that for users, we should respect >>>>>>>>>>> conventions >>>>>>>>>>>>>>>> over >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Conventions can be default values >>> explicitly >>>>>>>>>>>>> specified >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations, or they can be behaviors >>> that >>>>>>>>>>> follow >>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versions. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If the same code has different behaviors in >>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>> versions, >>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be a very bad thing. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree that for regular users, it is not >>>>>>>>> necessary >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the configurations related to Flink. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> By following conventions, they can have a >>>>> good >>>>>>>>>>>>>>>> experience. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let's get back to the practical situation >>> and >>>>>>>>>>>>> consider >>>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Case 1: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user is not familiar with the purpose >>> of >>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> table.optimizer.source.predicate-pushdown-enabled >>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> follows >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the convention of allowing predicate >>> pushdown >>>>>>> to >>>>>>>>>>> the >>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just understanding the >>>>>>>>>>>>> source.predicate-pushdown-enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and performing fine-grained toggle control >>>>> will >>>>>>>>>>> work >>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Case 2: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user understands the meaning of the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>> table.optimizer.source.predicate-pushdown-enabled >>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has set >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> its value to false. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We have reason to believe that the user >>>>>>>>> understands >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> meaning >>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicate pushdown configuration and the >>>>>>>>> intention >>>>>>>>>>>>> is to >>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown (rather than whether or not to >>> allow >>>>>>>>> it). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The previous choice of globally disabling >>> it >>>>> is >>>>>>>>>>>>> likely >>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> couldn't be disabled on individual sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From this perspective, if we provide more >>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support and provide detailed explanations >>> of >>>>>>> the >>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behaviors in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the documentation, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users can clearly understand the >>> differences >>>>>>>>>>> between >>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations and use them correctly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, I don't agree that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> table.optimizer.source.predicate-pushdown-enabled = >>>>>>>>>>>>> true >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source.predicate-pushdown-enabled = false >>>>> means >>>>>>>>>>> that >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> local >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration overrides the global >>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On the contrary, both configurations are >>>>>>>>>>> functioning >>>>>>>>>>>>>>>>>>>> correctly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The optimizer allows predicate pushdown to >>>>> all >>>>>>>>>>>>> sources, >>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can reject the filters pushed down by the >>>>>>>>>>> optimizer. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is natural, just like different >>>>> components >>>>>>>>> at >>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>> levels >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> responsible for different tasks. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The more serious issue is that if >>>>>>>>>>>>>>>>>>>>>>>>>>>> "source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does not respect >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>> "table.optimizer.source.predicate-pushdown-enabled”, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be invalidated. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This means that regardless of whether >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>> is >>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> true >>>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false, it will have no effect. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 22:24，Jane Chan < >>>>>>>>>>> qingyue....@gmail.com> >>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the in-depth clarification. >>> Here >>>>>>> are >>>>>>>>> my >>>>>>>>>>>>>>>> cents >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" are >>>>>>>>>>> configurations >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> components(optimizer and source >>> operator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We cannot assume that every user would be >>>>>>>>>>>>> interested in >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal components of Flink, such as the >>>>>>>>>>> optimizer >>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>> connectors, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific configurations associated with >>> each >>>>>>>>>>>>> component. >>>>>>>>>>>>>>>>>>>>>>> Instead, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be more concerned about knowing >>> which >>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>> enables >>>>>>>>>>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disables the filter push-down feature for >>>>> all >>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>> connectors, >>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter provides the flexibility to >>>>> override >>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>>>>>>> for a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> single >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source if needed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, from this perspective, I am inclined >>> to >>>>>>>>> divide >>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> based on the scope of their impact from >>> the >>>>>>>>> user's >>>>>>>>>>>>>>>>>>>> perspective >>>>>>>>>>>>>>>>>>>>>>>>>>>> (i.e. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> global-level or operator-level), rather >>> than >>>>>>>>>>>>>>>> categorizing >>>>>>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> component hierarchy from a developer's >>> point >>>>>>> of >>>>>>>>>>>>> view. >>>>>>>>>>>>>>>>>>>>>>> Therefore, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this premise, it is intuitive and natural >>>>> for >>>>>>>>>>> users >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand fine-grained configuration >>>>> options >>>>>>>>> can >>>>>>>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>>>>>>>>> global >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, if >>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>>> respect to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of "scan.filter-push-down.enabled" is >>>>> defined >>>>>>>>> as >>>>>>>>>>>>> true, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it means that just modifying >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" as >>>>>>>>>>>>>>>>>>>> false >>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effect, and filter pushdown will still be >>>>>>>>>>>>> performed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we define the default value of >>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false, it would introduce a difference in >>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>> compared >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <1>If I understand correctly, >>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>> is a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connector option, which means the only way >>>>> to >>>>>>>>>>>>> configure >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicitly >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify it in DDL (no matter whether >>> disable >>>>>>> or >>>>>>>>>>>>>>>> enable), >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> SET >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> command is not applicable, so I think it's >>>>>>>>> natural >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>>> respect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user's >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specification here. Otherwise, users might >>>>> be >>>>>>>>> more >>>>>>>>>>>>>>>>>> confused >>>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>> why >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DDL does not work as expected, and the >>>>> reason >>>>>>> is >>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "optimizer" configuration is set to a >>>>>>> different >>>>>>>>>>>>> value. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <2> From the implementation side, I am >>>>>>> inclined >>>>>>>>> to >>>>>>>>>>>>> keep >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter's >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> priority consistent for all conditions. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Let "global" denote >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and let "per-source" denote >>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source T, the following Truth table >>> (based >>>>> on >>>>>>>>> the >>>>>>>>>>>>>>>>>> current >>>>>>>>>>>>>>>>>>>>>>>>> design) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicates the inconsistent behavior for >>>>>>>>>>> "per-source >>>>>>>>>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>>>>>>>>>>>> global". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> .------------.---------------.------------------- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> ----.-------------------------------------. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | global | per-source | push-down for T >>> | >>>>>>>>>>>>> per-source >>>>>>>>>>>>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> global >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> :-----------+--------------+-----------------------+------------------------------------: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | true | false | false >>>>>>>>>>>>>>>>>> | Y >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> :-----------+--------------+-----------------------+------------------------------------: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | false | true | false >>>>>>>>>>>>>>>>>> | N >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> .------------.---------------.-----------------------.-------------------------------------. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 25, 2023 at 6:22 PM Jiabao >>> Sun < >>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Benchao for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand that the configuration of >>>>> global >>>>>>>>>>>>>>>>>> parallelism >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> task >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parallelism is at different granularities >>>>> but >>>>>>>>>>> with >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, >>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" are >>>>>>>>>>> configurations >>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> components(optimizer and source >>> operator). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From a user's perspective, there are two >>>>>>>>>>> scenarios: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Disabling all filter pushdown >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this case, setting >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to false is sufficient to meet the >>>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Disabling filter pushdown for specific >>>>>>>>> sources >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this scenario, there is no need to >>>>> adjust >>>>>>>>> the >>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Instead, the focus should be on the >>>>>>>>> configuration >>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" to meet >>> the >>>>>>>>>>>>>>>> requirement. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this case, users do not need to set >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" to >>>>>>>>>>>>>>>>>>>> false >>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> manually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable filter pushdown for specific >>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, if >>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>>> respect >>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of "scan.filter-push-down.enabled" is >>>>> defined >>>>>>>>> as >>>>>>>>>>>>> true, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it means that just modifying >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> "table.optimizer.source.predicate-pushdown-enabled" as >>>>>>>>>>>>>>>>>>>> false >>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effect, and filter pushdown will still be >>>>>>>>>>>>> performed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we define the default value of >>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false, it would introduce a difference in >>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>> compared >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The same SQL query that could >>> successfully >>>>>>> push >>>>>>>>>>>>> down >>>>>>>>>>>>>>>>>>>> filters >>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> old >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version but would no longer do so after >>> the >>>>>>>>>>>>> upgrade. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 17:10，Benchao Li < >>>>>>>>>>>>> libenc...@apache.org> >>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jiabao for the detailed >>>>> explanations, >>>>>>>>>>> that >>>>>>>>>>>>>>>>>> helps a >>>>>>>>>>>>>>>>>>>>>>>>> lot, I >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understand your rationale now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Correct me if I'm wrong. Your >>> perspective >>>>> is >>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>> "developer", >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means there is an optimizer and >>> connector >>>>>>>>>>>>> component, >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable this feature (pushing filters >>> down >>>>>>> into >>>>>>>>>>>>>>>>>>>> connectors), >>>>>>>>>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> must >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable it firstly in optimizer, and only >>>>>>> then >>>>>>>>>>>>>>>> connector >>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chance >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to decide to use it or not. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My perspective is from "user" that (Why >>> a >>>>>>> user >>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>> care >>>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> difference of optimizer/connector) , >>> this >>>>>>> is a >>>>>>>>>>>>>>>> feature, >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> way to control it, one way is to config >>> it >>>>>>>>>>>>> job-level, >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in table properties. What a user expects >>>>> is >>>>>>>>> that >>>>>>>>>>>>> they >>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>> control a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> feature in a tiered way, that setting it >>>>> per >>>>>>>>>>> job, >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine-grained tune it per table. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is some kind of similar to other >>>>>>>>> concepts, >>>>>>>>>>>>> such >>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parallelism, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users can set a job level default >>>>>>> parallelism, >>>>>>>>>>> and >>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tune it per operator. There may be more >>>>> such >>>>>>>>>>>>> debate >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> future >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e.g., we can have a job level config >>> about >>>>>>>>>>> adding >>>>>>>>>>>>>>>>>> key-by >>>>>>>>>>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> lookup >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> join, and also a hint/table property way >>>>> to >>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>> control >>>>>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> per lookup operator. Hence we'd better >>>>> find >>>>>>> a >>>>>>>>>>>>> unified >>>>>>>>>>>>>>>>>> way >>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> those similar kind of features. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao Sun <jiabao....@xtransfer.cn >>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>> 于2023年10月25日周三 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 15:27写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jane for further explanation. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> These two configurations correspond to >>>>>>>>>>> different >>>>>>>>>>>>>>>>>> levels. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" does not >>>>> make >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>> invalid. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The planner will still push down >>>>> predicates >>>>>>>>> to >>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Whether filter pushdown is allowed or >>> not >>>>>>> is >>>>>>>>>>>>>>>>>> determined >>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source's "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, >>>>> "table.optimizer.source.predicate" >>>>>>>>>>> does >>>>>>>>>>>>>>>>>>>> directly >>>>>>>>>>>>>>>>>>>>>>>>> affect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled”. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When the planner disables predicate >>>>>>> pushdown, >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> source-level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown will also not be executed, even >>> if >>>>>>> the >>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>> allows >>>>>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Whatever, in point 1 and 2, our >>>>> expectation >>>>>>>>> is >>>>>>>>>>>>>>>>>>>> consistent. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the 3rd point, I still think that >>> the >>>>>>>>>>>>>>>>>> planner-level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes precedence over the source-level >>>>>>>>>>>>> configuration. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may seem counterintuitive when we >>>>>>> globally >>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>> predicate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown but allow filter pushdown at the >>>>>>>>> source >>>>>>>>>>>>>>>> level. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 14:35，Jane Chan < >>>>>>>>>>>>> qingyue....@gmail.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for clarifying this. While by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher priority" I meant that this >>> value >>>>>>>>>>> should >>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> respected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whenever >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> set explicitly. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The conclusion that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. "table.optimizer.source.predicate" >>> = >>>>>>>>> "true" >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" = >>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Allow the planner to perform >>> predicate >>>>>>>>>>>>> pushdown, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>> individual >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources do >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not enable filter pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This indicates that the option >>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled = >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an individual source connector does >>>>> indeed >>>>>>>>>>>>> override >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> global-level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> planner settings to make a difference. >>>>> And >>>>>>>>>>> thus >>>>>>>>>>>>>>>> "has >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> priority". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> While for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. "table.optimizer.source.predicate" >>> = >>>>>>>>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Predicate pushdown is not allowed for >>>>> the >>>>>>>>>>>>> planner. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regardless of the value of the >>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, filter pushdown is >>>>>>> disabled. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this scenario, the behavior >>> remains >>>>>>>>>>>>> consistent >>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> old >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I still think >>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> respected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it is enabled for individual >>> connectors. >>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 25, 2023 at 1:27 PM Jiabao >>>>>>> Sun < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Benchao for the feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the current proposal, we >>> recommend >>>>>>>>>>> keeping >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" as >>>>>>> true, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and setting the the default value of >>>>>>> newly >>>>>>>>>>>>>>>>>> introduced >>>>>>>>>>>>>>>>>>>>>>>>> option >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" to >>> true >>>>>>> as >>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main purpose of doing this is to >>>>>>>>> maintain >>>>>>>>>>>>>>>>>>>> consistency >>>>>>>>>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versions, as whether to perform >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter pushdown in the old version >>>>> solely >>>>>>>>>>>>> depends >>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>> option. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That means by default, as long as a >>>>>>>>>>> TableSource >>>>>>>>>>>>>>>>>>>>>>> implements >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> SupportsFilterPushDown interface, >>>>> filter >>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> allowed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And it seems that we don't have much >>>>>>>>> benefit >>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>> changing >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of "table.optimizer.source.predicate" >>>>> to >>>>>>>>>>> false. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the priority of these two >>>>>>>>>>>>>>>> configurations, >>>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>>>>>>> believe >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes precedence over >>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exhibits >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> following behavior. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. >>> "table.optimizer.source.predicate" = >>>>>>>>>>> "true" >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" = >>>>> "true" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is the default behavior, >>> allowing >>>>>>>>> filter >>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. >>> "table.optimizer.source.predicate" = >>>>>>>>>>> "true" >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" = >>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Allow the planner to perform >>> predicate >>>>>>>>>>>>> pushdown, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>> individual >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sources do >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not enable filter pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. >>> "table.optimizer.source.predicate" = >>>>>>>>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Predicate pushdown is not allowed for >>>>> the >>>>>>>>>>>>> planner. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regardless of the value of the >>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration, filter pushdown is >>>>>>> disabled. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> In this scenario, the behavior >>> remains >>>>>>>>>>>>> consistent >>>>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> old >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> well. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From an implementation perspective, >>>>>>> setting >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> priority >>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>> higher >>>>>>> than >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" is >>>>>>>>>>>>> difficult to >>>>>>>>>>>>>>>>>>>>>>> achieve >>>>>>>>>>>>>>>>>>>>>>>>>>>> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because the >>>>>>>>> PushFilterIntoSourceScanRuleBase >>>>>>>>>>> at >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> planner >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> takes >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> precedence over the source-level >>>>>>>>>>>>>>>> FilterPushDownSpec. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Only when the >>>>>>>>>>> PushFilterIntoSourceScanRuleBase >>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> enabled, >>>>>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Source-level filter pushdown be >>>>>>> performed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, in my opinion, there >>>>>>> doesn't >>>>>>>>>>>>> seem to >>>>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> much >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> benefit in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> setting a higher priority for >>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It may instead affect compatibility >>> and >>>>>>>>>>>>> increase >>>>>>>>>>>>>>>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> complexity. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月25日 11:56，Benchao Li < >>>>>>>>>>>>>>>> libenc...@apache.org >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with Jane that fine-grained >>>>>>>>>>>>>>>> configurations >>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> priority than job level >>>>> configurations. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For current proposal, we can achieve >>>>>>> that: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Set >>>>>>> "table.optimizer.source.predicate" = >>>>>>>>>>>>> "true" >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> enable >>>>>>>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default, and set >>>>>>>>>>>>>>>> ""scan.filter-push-down.enabled" = >>>>>>>>>>>>>>>>>>>>>>>>> "false" >>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it per table source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Set >>>>>>> "table.optimizer.source.predicate" = >>>>>>>>>>>>>>>> "false" >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default, and set >>>>>>>>>>>>>>>> ""scan.filter-push-down.enabled" = >>>>>>>>>>>>>>>>>>>>>>>>> "true" to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it per table source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane Chan <qingyue....@gmail.com> >>>>>>>>>>>>> 于2023年10月24日周二 >>>>>>>>>>>>>>>>>>>>>>> 23:55写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe that the configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher priority at the planner >>> level >>>>>>>>> than >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it seems easy to implement >>> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Correct me if I'm wrong, but I >>> think >>>>>>> the >>>>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>> should >>>>>>>>>>> have a >>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>>>> priority >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> because >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> default value of >>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> true. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> As >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> turning off filter push-down for a >>>>>>>>> specific >>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> take >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> effect >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unless the default value of >>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> changed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to false, or, alternatively, let >>>>> users >>>>>>>>>>>>> manually >>>>>>>>>>>>>>>>>> set >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>> to >>>>>>>>> false >>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> selectively >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enable filter push-down for the >>>>> desired >>>>>>>>>>>>> sources, >>>>>>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>> less >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intuitive. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 24, 2023 at 6:05 PM >>>>> Jiabao >>>>>>>>> Sun >>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jane, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I believe that the configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "table.optimizer.source.predicate" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher priority at the planner >>> level >>>>>>>>> than >>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it seems easy to implement >>> now. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月24日 17:36，Jane Chan < >>>>>>>>>>>>>>>>>>>> qingyue....@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for driving this >>>>> discussion. I >>>>>>>>>>> have >>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>> small >>>>>>>>>>>>>>>>>>>>>>>>>>>> question >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "scan.filter-push-down.enabled" >>>>> take >>>>>>>>>>>>>>>> precedence >>>>>>>>>>>>>>>>>>>> over >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>> "table.optimizer.source.predicate" >>>>>>> when >>>>>>>>>>> the >>>>>>>>>>>>>>>> two >>>>>>>>>>>>>>>>>>>>>>>>> parameters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> might >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> conflict >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> each other? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jane >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 24, 2023 at 5:05 PM >>>>>>> Jiabao >>>>>>>>>>> Sun >>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks Jark, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we only add configuration >>>>> without >>>>>>>>>>>>> adding >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enableFilterPushDown >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method in the >>>>> SupportsFilterPushDown >>>>>>>>>>>>>>>> interface, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> each connector would have to >>>>> handle >>>>>>>>> the >>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>> logic >>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applyFilters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method to determine whether >>> filter >>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> needed. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This would increase complexity >>> and >>>>>>>>>>> violate >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> original >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applyFilters method. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On the contrary, we only need to >>>>>>> pass >>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly added enableFilterPushDown >>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to decide whether to perform >>>>>>> predicate >>>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think this approach would be >>>>>>> clearer >>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> simpler. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月24日 16:58，Jark Wu < >>>>>>>>>>>>> imj...@gmail.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi JIabao, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think the current interface >>> can >>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>> satisfy >>>>>>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> requirements. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The connector can reject all >>> the >>>>>>>>>>> filters >>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>> returning >>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> input >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filters >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as `Result#remainingFilters`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So maybe we don't need to >>>>>>> introduce a >>>>>>>>>>> new >>>>>>>>>>>>>>>>>>>> method to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown, but just introduce an >>>>>>>>> option >>>>>>>>>>>>> for >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connector. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jark >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, 24 Oct 2023 at 16:38, >>>>>>> Leonard >>>>>>>>>>> Xu >>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> xbjt...@gmail.com >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks @Jiabao for kicking off >>>>>>> this >>>>>>>>>>>>>>>>>> discussion. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Could you add a section to >>>>> explain >>>>>>>>> the >>>>>>>>>>>>>>>>>>>> difference >>>>>>>>>>>>>>>>>>>>>>>>>>>> between >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> connector level config >>>>>>>>>>>>>>>>>>>>>>>>> `scan.filter-push-down.enabled` >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> query >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> level config >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> `table.optimizer.source.predicate-pushdown-enabled` ? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Leonard >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2023年10月24日 下午4:18，Jiabao >>> Sun < >>>>>>>>>>>>>>>>>>>>>>>>>>>> jiabao....@xtransfer.cn >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .INVALID> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Devs, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a >>>>>>> discussion >>>>>>>>> on >>>>>>>>>>>>>>>>>>>> FLIP-377: >>>>>>>>>>>>>>>>>>>>>>>>>>>> support >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disable filter pushdown for >>>>>>>>> Table/SQL >>>>>>>>>>>>>>>>>>>> Sources[1]. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently, Flink Table/SQL >>> does >>>>>>> not >>>>>>>>>>>>> expose >>>>>>>>>>>>>>>>>>>>>>>>>>>> fine-grained >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users to enable or disable >>>>> filter >>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, filter pushdown has >>>>> some >>>>>>>>>>> side >>>>>>>>>>>>>>>>>>>> effects, >>>>>>>>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> additional >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> computational pressure on >>>>> external >>>>>>>>>>>>> systems. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Moreover, Improper queries >>> can >>>>>>> lead >>>>>>>>>>> to >>>>>>>>>>>>>>>>>> issues >>>>>>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> full >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> scans, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which in turn can impact the >>>>>>>>> stability >>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>> external >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> systems. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Suppose we have an SQL query >>>>> with >>>>>>>>> two >>>>>>>>>>>>>>>>>> sources: >>>>>>>>>>>>>>>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> database. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The database is sensitive to >>>>>>>>>>> pressure, >>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configure >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not perform filter pushdown to >>>>> the >>>>>>>>>>>>> database >>>>>>>>>>>>>>>>>>>>>>> source. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, we still want to >>>>> perform >>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>> pushdown >>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> source >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decrease network IO. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I propose to support >>>>>>> configuration >>>>>>>>> to >>>>>>>>>>>>>>>>>> disable >>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> push >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> down for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Table/SQL sources to let user >>>>>>> decide >>>>>>>>>>>>>>>> whether >>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>> perform >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> pushdown. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to your >>>>> feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiabao >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benchao Li >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Benchao Li >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>>
