adriangb commented on issue #16200: URL: https://github.com/apache/datafusion/issues/16200#issuecomment-2930710421
Yeah I see two ways to go about that: 1. We make individual wrappers for each operation that decide how to fetch things. A MetadataLoader, FileReaderFactory, etc. I think this is the direction things are headed in now. 2. Configs/settings like Postgres' `random_page_cost` or some sort of `target_fetch_size`, etc and we have DataFusion determine what to do with that information. We could even add the ability to dynamically measure and set that so that eg DataFusion CLI can adapt itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
