jizezhang commented on issue #19573: URL: https://github.com/apache/datafusion/issues/19573#issuecomment-3706681646
Hi @alamb @BlakeOrth I was experimenting with ways to make the cache table scoped, and would like to get some high level feedback from you. I have a draft PR https://github.com/apache/datafusion/pull/19616 for one approach I considered, which is to continue using a session level cache as is currently but make the cache key table scoped. The PR contains a short description and some cli testing results to verify that. Another approach I tried is to attach an independent cache to each table (e.g. add a field in `ListingTable` struct) and have a session level map from table reference to table cache to track and display at session level. However as I was experimenting, I found it a bit tricky to handle user-provided `ListFileCache` implementation, which is currently given as a `Arc<dyn ListFilesCache>`. To support independent table scoped cache, I think we may need some kind of `ListFilesCacheFactory` instead? Please let me know your thoughts/suggestions, and correct me if anything does not make sense. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
