phillipleblanc commented on issue #15173: URL: https://github.com/apache/datafusion/issues/15173#issuecomment-2846043563
Yeah that makes sense. Part of the complexity here is that several of the features needed to make the ListingTableProvider work today (i.e. partition columns) are actually implemented in the core FileStream. And in order to add the metadata columns I want, I need to pipe through the ObjectMeta in the FileStream as well. So it would end up being quite a lot of code duplication to fork out of the core repo to get this working as is. I do think there is opportunity for some better extensibility/abstractions that would allow people to implement something like this on their own - but I'm not sure what that is yet. I think a good first step would be figuring out how to abstract out the partition columns so that the existing ListingTable can work without any special code in core. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
