alamb commented on PR #10745:
URL: https://github.com/apache/datafusion/pull/10745#issuecomment-2175817937
> Hi @alamb, is there any update on this topic? Or maybe I can help by
splitting out some traits into the new crate `datafusion-catalog` first?
Hi @goldmedal -- sorry for the lack of update. From my perspective, it is
not a good idea to add any more dependencies (even optional) to the core
datafusion crate.
After thinking about this some more, I think there is a difference between:
1. The logic that interprets table names as potential object store locations
2. The logic that handles urls like `s3://...` by automatically registering
object store factories
Putting the first in the datafusion core makes sense. I think the second
comes with many dependencies and thus doesn't make sense.
Thus, what do you think about somehow splitting the two pieces of
functionalty apart? Maybe via a trait like
```rust
struct DynamicTableProvider {
...
/// A callback function that is
url_lookup: Arc<dyn UrlLookup>
}
/// Trait for looking up the correct object store instance based on url
pub trait UrlLookup{
fn lookup(&self, url: &Url) -> Result<Arc<dyn ObjectStore>>;
}
```
Thank you for working on this PR / IDE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
