alamb commented on PR #13424:
URL: https://github.com/apache/datafusion/pull/13424#issuecomment-2494584685
> Can you expand on point 1? My naive expectation was that all network io
went through the main runtime.
Yes, that is what should happen. The problem is here. As written, calling
`IoObjectStore::list ` will actually try and do IO on the wrong thread pool (as
the stream executes on the wrong runtime)
```rust
impl ObjectStore for IoObjectStore {
fn list(&self, prefix: Option<&Path>) -> BoxStream<'_,
Result<ObjectMeta>> {
// TODL run the inner list on the dedicated executor
let inner_stream = self.inner.list(prefix);
inner_stream
}
```
I am trying to figure out how to make Rust run the stream on the right
threadpool (we need to wrap it somehow and transfer results). This is what
@tustvold was concerned about here:
https://github.com/datafusion-contrib/datafusion-dft/pull/248#issuecomment-2489058438
I thought I could use the `CrossRTStream` added in this PR, Ideally it would
look like
```rust
impl ObjectStore for IoObjectStore {
fn list(&self, prefix: Option<&Path>) -> BoxStream<'_,
Result<ObjectMeta>> {
let inner_stream = self.inner.list(prefix);
// run the inner list on the dedicated executor
self.dediated_executor.wrap_stream(inner_stream, convert_error)
}
```
But but I am fighting Rust compiler lifetime shenanigans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
