alamb commented on code in PR #21327:
URL: https://github.com/apache/datafusion/pull/21327#discussion_r3053630852
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -454,32 +453,46 @@ impl Future for ParquetOpenFuture {
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) ->
Poll<Self::Output> {
loop {
- // If waiting on IO, poll
+ // If planner I/O completed, resume with the returned planner.
if let Some(io_future) = self.pending_io.as_mut() {
- ready!(io_future.poll_unpin(cx))?;
+ let maybe_planner = ready!(io_future.poll_unpin(cx));
+ // Clear `pending_io` before handling the result so an error
+ // cannot leave both continuation paths populated.
self.pending_io = None;
+ self.planner = Some(maybe_planner?);
Review Comment:
yes I thik you are right. It would be possible to have a ready planner in
theory with the morsel API (but not with the current parquet opener
implementation)
I started over thinking this and adding an `enum` but then I realized the
parquet opener can never return more than one planner, so I think this can not
be hit.
I'll add the assert
##########
datafusion/datasource-parquet/src/opener.rs:
##########
@@ -454,32 +453,46 @@ impl Future for ParquetOpenFuture {
fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) ->
Poll<Self::Output> {
loop {
- // If waiting on IO, poll
+ // If planner I/O completed, resume with the returned planner.
if let Some(io_future) = self.pending_io.as_mut() {
- ready!(io_future.poll_unpin(cx))?;
+ let maybe_planner = ready!(io_future.poll_unpin(cx));
+ // Clear `pending_io` before handling the result so an error
+ // cannot leave both continuation paths populated.
self.pending_io = None;
+ self.planner = Some(maybe_planner?);
Review Comment:
in 696a42dce0
##########
datafusion/datasource/src/mod.rs:
##########
@@ -38,6 +38,7 @@ pub mod file_scan_config;
pub mod file_sink_config;
pub mod file_stream;
pub mod memory;
+pub mod morsel;
Review Comment:
in 556690f0af383451cd97175db285a90daa35f4cf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
