alamb commented on PR #23492: URL: https://github.com/apache/datafusion/pull/23492#issuecomment-5273121874
I have been thinking about this PR and how it could play into our "turn pushdown on by default" story We have put a lot of work in upstream to make applying filters during parquet decoding faster (thanks to @hhhizzz @haohuaijin and many others). However, enabling filter pushdown today **also** changes the I/O patterns (so that the all the needed columns are not fetched up front, but instead are fetched sequentially) I think one thing we could do with a nice I/O pre-fetching abstraction would be to start a request to fetch bytes for all columns required in a single request, so the I/O pattern is the same -- and then just enable the pushdown filter application once we already had the data 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
