alamb commented on issue #10782: URL: https://github.com/apache/datafusion/issues/10782#issuecomment-2473223198
> Oh i see, should we be putting priority on this soon? or should this be visited back another time? In my opinion, this would qualify as "Things that make DataFusion easier to work with" (details [here](https://github.com/apache/datafusion/issues/13386#issue-2654282525)) and thus I will also try and find time to help this project along (both reviews as well as project planning) I have been annoyed recently at the amount of time it takes to compile datafusion > For example, if we want to move CsvExec out, we’ll also need to move FileScanConfig and FileCompressionType. To extract FileScanConfig, we’ll need to move PartitionedFile as well. This process will reveal that we must move the entire datasource::listing module first, and whenever we need SessionState, replace it with trait Session > I went through the thread and I couldn't figure out if we had identified/agreed on a set of issues that compose this epic @alamb Here is what I suggest: 1. Try and complete moving the items listed in this issues description. 1. Do NOT try and move `ListingTableProider` as part of this project (that is big enough that I think we should make its own epic for the reasons @jonathanc-n mentions) I took a quick look and I think there are two things we could move to `datafusion-catalog` to complete this ticket: 1. The `Memory*` providers: https://github.com/apache/datafusion/blob/4c1ec807fb70ef8abe96299760dd2faaa556a49c/datafusion/core/src/catalog_common/memory.rs 2. The `InformationSchema` providers: https://github.com/apache/datafusion/blob/4c1ec807fb70ef8abe96299760dd2faaa556a49c/datafusion/core/src/catalog_common/information_schema.rs Perhaps we can file tickets for those items and I can link them to this epic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
