gabotechs commented on code in PR #127:
URL: 
https://github.com/apache/datafusion-iceberg/pull/127#discussion_r4640538855


##########
crates/datafusion/src/read/file_scan_task_provider/lazy.rs:
##########
@@ -36,19 +38,26 @@ use crate::to_datafusion_error;
 /// Snapshot summary field from the spec's "Optional Snapshot Summary Fields".
 const TOTAL_RECORDS: &str = "total-records";
 
-/// Discovers files while executing, in a single partition: the files of a
-/// snapshot that may match a predicate.
+/// Plans file tasks during execution. Each output partition reads from the
+/// same task stream; unopened partitions do not reserve files. Dropping the
+/// last reader drops the stream, and late readers see EOF.
+///
+/// Opening another reader does not restart the scan. Use DataFusion's
+/// `reset_plan_states` before executing the same parallel plan again.
 #[derive(Debug, Clone)]
 pub struct LazyFileScanTaskProvider {
     table: Table,
     schema: SchemaRef,
     snapshot_id: Option<i64>,
     projection: Vec<String>,
     predicate: Option<Predicate>,
+    target_partitions: usize,
+    // Only used when callers open the provider directly, without 
DataSourceExec.
+    direct_tasks: Arc<OnceLock<SharedFileTasks>>,

Review Comment:
   Need to read a bit more, but this does not look right. Having fields that 
are only used when this is used in a specific way... it seems to be leaking 
details about how it's used, but this struct should not matter.



##########
crates/datafusion/src/read/file_scan_task_provider/provider.rs:
##########
@@ -53,6 +53,14 @@ pub trait FileScanTaskProvider: Any + Debug + Send + Sync {
         context: FileScanTaskProviderContext,
     ) -> Result<iceberg::scan::FileScanTaskStream>;
 
+    /// Optionally supplies a single task stream for all partitions of one 
execution.
+    /// The source lets each partition pull the next unclaimed task from that 
stream.
+    /// Return a fresh stream on each call; defer I/O until it is polled.
+    /// The default uses [`Self::open`] for each partition instead.
+    fn open_shared(&self) -> Option<iceberg::scan::FileScanTaskStream> {
+        None
+    }

Review Comment:
   This does not look right. We should avoid having "optional" methods that are 
only relevant for 1 specific case in a trait that claims to be generic. 
Ideally, we can keep this trait as "proxy" as possible to the DataSource, 
without inventing new methods that do not exist there.
   



##########
crates/datafusion/src/read/scan.rs:
##########
@@ -327,6 +348,14 @@ impl DataSource for IcebergDataSource {
         self.fetch
     }
 
+    fn create_sibling_state(
+        &self,
+        _config: &ConfigOptions,
+    ) -> Option<Arc<dyn Any + Send + Sync>> {
+        let tasks = self.file_scan_task_provider.open_shared()?;
+        Some(Arc::new(SharedFileTasks::new(tasks)))
+    }

Review Comment:
   This is weirdly threaded. The idea behind the sibling state introduced in 
https://github.com/apache/datafusion/pull/21351 is that the returned `Any` type 
is what owns the shared state, however, even if we do are consuming the sibling 
state API, we are anyways storing state in the structs owned by the plan:
   
   ```rust
   direct_tasks: Arc<OnceLock<SharedFileTasks>>,
   ```
   
   I'd either:
   - Not use the sibling state API at all
   - Fully delegate statefulness to the sibling state
   
   But not something in between.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to