jayshrivastava commented on issue #21207: URL: https://github.com/apache/datafusion/issues/21207#issuecomment-5034576911
> So, FWIW: when we consume InList (e.g. "small") filters in our scans, we frequently tear them apart in order to evaluate them using an index, rather than using Arrow. > If a dynamic filter has a lot of consumers then maybe it might make sense to execute these types of merges eagerly... but otherwise, letting the consumer decide whether it wants to further "optimize" it (concatenating, merging, etc) before applying it would be good for us. On this topic, for certain data sources, it's difficult to translate and pushdown `CASE hash(expr)`. For example, if the data source is a an API endpoint implemented in a different language or a some SQL database, it's the `hash` function is not easily translated. For these cases, I think the `CASE hash` should be opt-in or easy removable (ex. we currently have a merge() implementation which merges all per-partition expressions into a range + in list). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
