jeffreyssmith2nd commented on code in PR #14028:
URL: https://github.com/apache/datafusion/pull/14028#discussion_r1905736433
##########
datafusion/core/src/datasource/physical_plan/file_stream.rs:
##########
@@ -478,7 +478,12 @@ impl<F: FileOpener> FileStream<F> {
reader,
)),
partition_values,
- }
+ };
+ // Return control to the runtime
when we're ready to open the next file
+ // to prevent uncancellable
queries in scenarios with many large files.
+ // This functions similarly to a
`tokio::task::yield_now()`.
+ cx.waker().wake_by_ref();
+ return Poll::Pending;
Review Comment:
I thought it was odd too (I had seen this pattern elsewhere) but looking at
the implementation of `yield_now`, when you `poll` it does a `context::defer`
and then returns `Pending`
[(source)](https://github.com/tokio-rs/tokio/blob/bd3e8577377a2b684b50fc0cb50d98f03ad09703/tokio/src/task/yield_now.rs#L57-L59).
`context::defer` will directly call `wake_by_ref` if called from outside of
the runtime
[(source)](https://github.com/tokio-rs/tokio/blob/bd3e8577377a2b684b50fc0cb50d98f03ad09703/tokio/src/runtime/context.rs#L167-177).
I could be misunderstanding what "outside the runtime" means in this
case/how that will actually interact with the yield.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
