jeffreyssmith2nd commented on code in PR #14028:
URL: https://github.com/apache/datafusion/pull/14028#discussion_r1907311925
##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -478,7 +478,12 @@ impl<F: FileOpener> FileStream<F> {
reader,
)),
partition_values,
- }
+ };
+ // Return control to the runtime
when we're ready to open the next file
+ // to prevent uncancellable
queries in scenarios with many large files.
+ // This functions similarly to a
`tokio::task::yield_now()`.
+ cx.waker().wake_by_ref();
+ return Poll::Pending;
Review Comment:
If its clearer, we can remove these lines and change the lines above it to
```
self.state = FileStreamState::Open {
future: Box::pin(async {
yield_now().await;
reader
}),
partition_values,
};
```
This works because this future gets polled in the next iteration of the loop
when we transition to the Open state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
