jonahgao commented on issue #13994: URL: https://github.com/apache/datafusion/issues/13994#issuecomment-2574295463
> One last question: If I have a query with [filterA,filterB] and on the initial call I return [Unsupported,Exact] and on a subsequent call you send me [filterA] and I return [Unsupported], then I believe the scan filters comes through as [] (empty). Is DataFusion still respecting the Exact on filterB? My guess is not, that is another important problem as if my custom data source only sends back data that matches filterB and DataFusion is again filtering all result data by filterB again the pushdown is not doing anything. I see this now, it does work because my data source did filter the data properly, but my guess is DataFusion is doing extra work it should not do. In the current implementation, returning Exact filters will be pushed down to the TableScan and will be retained. Each round of pushdown will not revoke the previous effect. But it's best not to rely on this; `supports_filters_pushdown` is a guarantee for DataFusion that your data source has the ability to handle a certain filter. DataFusion can push it down, but that doesn't mean it must be pushed down. Your scan needs to work regardless of whether filters are present. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
