alamb commented on issue #24288: URL: https://github.com/apache/datafusion/issues/24288#issuecomment-5268764391
I think we should likely start with the appropriate API in the arrow-rs parquet crate (MetadataPushDecoder) so we can specify what columns to read (I don't think there is any way to do that now). Once we have the API in arrow, I think it will be straightforward to integrate it into DataFusion I did a recent PR here to avoid reloading the page index when it wasn't present. I think we could extend the same idea - https://github.com/apache/datafusion/pull/24150 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
