ratuldawar11 commented on issue #23263: URL: https://github.com/apache/datafusion/issues/23263#issuecomment-4900303686
> datafusion.execution.parquet.pushdown_filters = true (default is false). This enables Parquet RowFilter, which decodes the filter column (URL) first, applies LIKE '%google%', and only decodes the other 104 columns for surviving rows within each row group. For ~1% match rate this eliminates the wide-decode on ~99% of rows. The local run on this looks very promising, but this will benefit for filter only cases right ? Suppose a query ``` SELECT * FROM hits ORDER BY "EventTime" LIMIT 10; ``` This won't help with the filter pushdown but yes the TopK dynamic row group pruning should be really helpful here and I think since you tested with master branch that also was used here. Then we should set this property as default right ? Any reason not to ? This should be helpful in many other cases too not just wide selections as this completely skips rows, right ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
