ion-elgreco commented on PR #14286: URL: https://github.com/apache/datafusion/pull/14286#issuecomment-2707431726
> After reading all the relevant feedback on this PR here is my summary: > 1. Some people have found it valuable to have an ObjectStore which does I/O on a different tokio Runtime > 2. The DedicatedExecutor API seems overly complicated for what it is doing. > > Thus my suggested next steps are: > 1. For the near term / workaround, break out the object store wrapper to pass work on different runtimes . Perhaps based on the [one in delta-rs](https://github.com/delta-io/delta-rs/blob/e30ab7e366eb209718c87acb6974a815503181bc/crates/core/src/storage/mod.rs#L116-L120) > 2. Update this example to that wrapper and use a tokio::Runtime directly rather than a DedicatedExecutor > 3. Work on a more general purpose IO API (see https://github.com/apache/datafusion/issues/14854 from @Xuanwo) which might make it easier to get this right without the shim / spawn io Sounds like a prime candidate for that object-store-utils crate :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
